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ABSTRACT 

The  purpose  of  this  study  is  to  create  a  guide  for  technical  review  board  chairperson 
conducting  technical  review  boards  for  rocket  testing  performed  by  the  Air  Force  Research 
Laboratory’s  Space  Missile  Propulsion  Division  located  at  Edwards  Air  Force  Base  in 
California.  Technical  review  boards  are  independent  reviews  of  the  test  programs,  providing 
a  crucial  check  and  balance  in  the  programs  overall  systems  engineering  and  quality  process. 
As  will  be  discussed  in  the  subsequent  pages,  current  technical  review  boards  are  inefficient 
and  of  inconsistent  quality. 

The  importance  of  a  high  quality,  efficient  technical  review  has  never  been  more 
important  due  to  the  modern-era  challenges  relating  to  the  limited  hands-on  experience  and 
fewer  number  of  current  rocket  testers.  Thus  a  paradigm  shift  in  the  way  technical  review 
boards  are  conducted  at  AFRL  for  rocket  test  programs  is  needed.  The  local  guidance 
created  from  this  study  is  intended  to  assist  in  the  execution  of  technical  review  boards. 

The  guidance  created  by  this  study  can  assist  the  technical  review  board  chairperson 
in  running  a  more  efficient  and  effective  review.  The  guidance  includes  lessons  learned 
about  complexity  of  organizational  decision  making,  policies,  procedures,  checklist, 
organizational  cultural  change,  quality  assurance,  and  meeting  management.  In  addition,  the 
checklist  created  as  part  of  this  guidance  will  assist  in  making  sure  the  technical  review  board 
chairperson  does  not  overlook  a  critical  topic  for  review. 
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CHAPTER  1 
Introduction 

Introduction  and  Research  Question 

The  Air  Force’s  mission  is  to  protect  the  United  States  from  threats.  While  it  is 
preferable  to  provide  this  defense  as  deterrent,  the  Air  Force  is  always  ready  to  fight  for  our 
freedom  when  necessary.  Advanced  weapons  systems  used  by  expertly  trained  Airman  are  a 
key  element  to  the  Air  Force’s  success  in  a  fight  and  therefore  one  of  its  greatest  assets  as  a 
deterrent  to  war.  In  order  to  remain  the  most  advanced  Air  Force  in  the  world,  this 
technology  must  be  constantly  improved  upon  and  made  available  to  the  warfighter. 
According  to  Remen  (2000),  “The  Air  Force  is  always  seeking  to  be  a  better  warfighter,  to 
always  be  ready  and  able  to  win  if  called  to  war,  ensuring  its  systems  are  the  best  in  the 
world.”  (p.  1) 

Creating  advanced  technology  requires  vision  and  experimentation.  Wiswell  and 
Huggins  noted  (1990), 

“New  weapon  systems  generally  rely  on  technology  developed  by  the  laboratories 
which  begins  10  to  15  years  ahead  of  full-scale  development.  It  is  the  laboratories 
innovative  science  and  technology  program  that  is  the  key  to  providing  an  affordable, 
qualitatively  superior  military  force.”  (p.  7) 

Continuously  having  the  most  advanced  technology  available  requires  long  term  commitment 
to  research  and  development.  The  Air  Force  Research  Laboratory  (AFRL)  is  the 
embodiment  of  the  Air  Force’s  determination  to  have  the  most  superior  weapons  systems 
available  at  all  times.  Remen  wrote  (2000), 
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“The  Air  Force  Research  Laboratory  is  the  Air  Force’s  arm  for  conducting  this 
research  and  development.  AFRL  has  a  long  history  of  developing  technologies  to 
address  the  warfighter’s  needs  and  a  long  history  of  having  those  technologies 
leveraged  or  even  outright  used  by  both  NASA  and  Industry  and  applied  to  their 
mission  needs.”  (p.  5) 

Remen  observed  (2000),  “The  military  has  always  been  the  leader,  first  in  Air  then  in 
Space,  pushing  what  could  be  done  in  space.”  (p.  3)  Early  technical  success  of  the  Air  Force 
Research  Laboratory  demonstrations  provided  the  confidence  required  to  proceed  with  the 
development  of  the  Space  Shuttle  Main  Engine  (Wiswell  &  Huggins,  1990,  p.  5)  According 
to  Remen  (2000),  “Every  existing  US  space  launch  system,  commercial  and  military,  was 
derived  from  Air  Force  developed  systems.”  (p.  3)  However,  the  Air  Force  can’t  afford  to  be 
content  with  past  successes  and  must  continue  to  build  for  the  future.  Wiswell  and  Huggins 
declared  (1990),  “The  laboratory  is  and  will  continue  to  fulfill  its  responsibility  for 
developing  technologies  for  future  space  and  missile  systems  of  the  Air  Force.”  (p.  6) 

The  Space  Missile  Propulsion  Division  of  AFRL  at  Edwards  Air  Force  Base  has  a 
long  and  proud  history  and  is  a  world  leader  for  the  testing  of  rockets.  Wiswell  and  Huggins 
reported  (1990),  “The  Astronautics  Lab  started  on  Luehman  Ridge  east  of  the  Edwards  base 
as  a  part  of  the  Air  Force  Power  Plant  Laboratory  at  Wright-Patterson  AFB  in  Ohio  for  test 
and  evaluation  work.”  (p.  1)  In  1952  the  first  facilities  were  completed  and  in  1959  the 
Power  Plant  Laboratory,  Rocket  Propulsion  Branch  was  moved  to  Edwards  (Wiswell  and 
Huggins,  p.  1).  The  Space  Missile  Propulsion  Division  has  gone  by  many  different  names 
during  its  existence,  but  no  matter  the  name  the  Space  Missile  Propulsion  Division  has  been 
at  the  forefront  of  breakthroughs  in  rocket  technology.  The  Space  Missile  Propulsion 
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Division  has  contributed  to  the  success  of  the  Saturn  5,  Space  Shuttle,  Atlas,  Titan, 
Minuteman,  and  Peacekeeper  programs  to  name  a  few. 

With  a  tradition  of  success  on  complex  cutting  edge  technology  programs  it  is  easy 
for  an  organization  to  get  lulled  into  a  sense  of  complacency.  But  there  are  new  challenges 
that  have  emerged  and  many  of  them  are  not  research  related.  One  of  these  is  the  decreasing 
workforce  available  to  perform  all  the  various  tasks  required  to  meet  the  challenges  of  highly 
complex  technical  work  involved  in  rocket  testing.  Additionally,  a  large  part  of  the  corporate 
knowledge  base  created  during  the  50’s  and  60’s,  the  glory  days  of  the  rocket  industry,  has 
disappeared  because  of  retirement.  With  a  significant  reduction  in  rocket  programs  and  the 
number  of  rocket  tests  each  year,  the  new  generation  of  rocket  testers  is  finding  fewer 
learning  opportunities.  This  has  resulted  in  test  programs  being  performed  by  fewer  and 
often  less  experienced  testers  than  previous  generations,  which  has  increased  the  importance 
of  the  Technical  Review  Board  (TRB)  process.  This  study  will  look  at  the  TRB  process  in 
order  to  find  ways  to  improve  it. 

Technical  review  boards  are  independent  evaluations  of  the  test  programs,  providing 
a  crucial  check  and  balance  in  the  program’s  overall  systems  engineering  and  quality  process. 
The  Air  Force  Research  Laboratory  conducts  technical  review  boards  to  reduce  risks  to 
programs  and  personnel.  AFRL  is  interested  in  improving  the  TRB  process  as  part  of  the 
laboratory’s  continuous  quality  improvement  program  to  assure  future  test  program  success 
by  adapting  to  a  changing  environment  and  incorporating  lessons  learned.  Air  Force 
regulations  are  vague  about  how  TRBs  are  to  successfully  evaluate  the  technical  aspects  of  a 
program.  Current  TRB  reviews  are  inefficient  and  the  quality  of  the  review  provided  is 
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inconsistent  from  one  TRB  to  the  next.  Can  detailed  local  TRB  guidance  be  created  that  will 
improve  the  quality  and  efficiency  of  TRB  reviews  for  rocket  test  programs  at  AFRL? 

Brief  Background 

Air  Force  regulations  require  test  programs  to  have  technical  review  boards.  These 
reviews  are  intended  to  provide  greater  confidence  the  test  program  will  succeed.  Due  to  the 
expense  of  tests  and  inherent  dangers  associated  with  testing  rockets.  Air  Force  management 
places  a  high  level  of  importance  on  technical  review  boards  being  conducted.  Because  of 
the  small  margin  for  error  and  potential  for  catastrophic  failure,  quality  technical  reviews  are 
crucial  to  continuing  the  high  level  of  success  test  programs  have  seen  at  AFRL. 

The  Space  Missile  Propulsion  Division  has  to  comply  with  four  levels  of  guidance  on 
technical  review  boards,  based  on  its  placement  in  the  Air  Force  hierarchy.  The  Air  Force  is 
split  into  different  commands.  AFRL  is  the  laboratory  organization  of  the  Air  Force 
Material  Command  (AFMC).  AFRL  is  broken  into  directorates,  which  are  further  broken 
down  into  divisions.  The  Space  Missile  Propulsion  Division  is  part  of  the  AFRL  Propulsion 
Directorate.  The  Air  Force,  AFMC,  AFRL,  and  the  Edwards  portion  of  the  AFRL 
Propulsion  Directorate,  AFRL/RZ-West,  all  have  guidance  created  for  TRBs. 

The  four  regulations  flow  from  the  highest  level  organization  down,  and  therefore 
there  is  a  large  amount  of  repetition  in  the  wording.  Select  portions  of  each  regulation’s 
wording  has  been  included  here  in  order  to  highlight  what  the  regulations  include  without 
boring  the  reader  with  redundancy.  The  highest  level  regulation  is  the  Air  Force  Instruction 
99-103,  Capabilities-Based  Test  and  Evaluation  which  states  the  following  related  to 
technical  review  boards  (2009),  “The  TRB  assesses  the  soundness  of  system  design  and  test 
plans  to  reduce  test  risk.  ”  (p.  45)  The  instruction  does  not  provide  any  details  about  how  to 
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assess  the  soundness  of  the  designs  or  test  plans,  but  it  does  state  a  goal  which  is  to  reduce 
test  risk.  The  next  level  of  regulation  is  the  AFMC  Instruction  99-103,  Test  Management, 
which  states  (2004),  “The  technical  review  will  verify  that  the  overall  method  of  test  and  test 
data  acquisition  is  adequate  to  evaluate  the  requirements  and  to  verify  objectives  can  be  met 
with  acceptable  risk.”  (p.  4)  The  AFMC  instruction  provides  a  little  more  direction  on  the 
goal  of  reducing  risk  by  stating  that  objective  can  be  met  with  acceptable  risk,  although  what 
is  acceptable  is  not  clarified.  The  AFMC  regulation  also  fails  to  provide  details  about  how  to 
go  about  verifying  the  overall  method  of  test  and  test  data  acquisition  is  adequate  to  evaluate 
the  requirements.  The  third  regulation  is  the  AFRL  Instruction  61-103,  AFRL  Research  Test 
Management,  and  it  states  (2007),  “As  a  minimum,  technical  reviews  will  assess  test 
requirements,  techniques,  approaches,  schedules,  and  objectives.”  (p.  11)  The  AFRL  level 
instruction  provides  a  small  amount  of  insight  for  the  review  board  member  of  what  they  are 
to  assess,  but  does  not  provide  any  criteria  for  the  assessment.  The  final  regulation  is  the 
AFRL/RZ-West  Operating  Instruction  61-103,  Research  and  Development  Test  Operations, 
which  states  (2010), 

“1 1.0  Technical  Review  Boards  (TRB):  The  TRB  is  responsible  for  reviewing  the 
technical  soundness  of  the  facility  systems  design,  verification  and  validation  plan, 
test  plans,  NCARS,  procedures,  and  test  readiness  data  to  evaluate  if  the  program  will 
meet  the  test  requirements  and  program  risk. . .”  (p.  9) 

The  AFRL/RZ-West  operating  instruction  provides  details  where  to  find  the  infonnation  to 
be  evaluated  and  includes  reference  to  the  verification  and  validation  plan  (V&V  plan).  The 
V&V  plan  and  its  importance  will  be  discussed  further  in  chapters  2  and  4. 
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As  can  be  seen,  all  four  Air  Force  regulations  which  define  TRBs  are  vague  about 
how  to  perform  a  successful  review  or  about  the  exact  goals  of  the  review.  Qualitative 
words  like  “soundness”,  “method”,  “adequate”,  and  “acceptable”  are  used.  This  vagueness 
is  intentional  due  to  the  variety  of  different  technical  programs  the  Air  Force  conducts  and 
the  need  for  flexibility  due  to  these  differences.  Even  within  programs  solely  within  the 
Space  Missile  Propulsion  Division,  there  is  a  large  range  of  types,  to  include  the  testing  of 
large  rocket  motors,  new  propellants,  or  advanced  materials  to  name  a  few.  Despite  these 
diversity  on  can  find  a  large  number  of  similarities  between  rocket  test  programs,  which 
would  justify  having  a  single  standard  TRB.  This  paper  will  explore  TRBs  for  rockets  tested 
by  the  Space  Missile  Propulsion  Division. 

Two  of  the  main  competing  values  related  to  TRBs  are  thoroughness  and  efficiency. 
The  challenge  for  any  test  program  is  finding  the  point  where  attention  to  detail  is  sufficient 
and  must  give  way  to  schedule  considerations.  The  rockets  being  tested  are  often 
multimillion  dollar  assets  and  test  failure  could  lead  to  loss  of  the  asset  or  the  desired  data. 

At  the  same  time  schedule  delays  can  cost  thousands  of  dollars  a  day  to  retain  defense 
contractor  expertise  or  by  delaying  a  launch  that  is  dependent  on  the  test  results.  In  the  past, 
thorough  TRBs  have  required  a  large  investment  in  time  and  resource.  One  test  program  had 
TRB  boards  that  meet  once  or  twice  a  week  for  6  months.  Another  test  program  chose  to 
relocate  to  another  test  facility  due  to  the  $75,000  estimate  for  the  TRB,  which  was 
approximately  a  third  of  the  total  test  costs.  Proponents  of  thorough  TRBs  are  concerned 
with  technical  issues  not  being  missed  because  of  the  inherent  dangers  and  large  costs 


associated. 
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TRBs  that  have  been  considered  efficient  have  been  defined  so  because  they  required 
small  investments  in  time  and  resources.  Proponents  of  efficient  TRBs  are  concerned  with 
available  resources  and  schedules.  This  concern  is  increased  because  resources,  especially 
personnel,  are  shared  between  programs.  It  is  my  opinion,  a  balance  must  be  found  between 
the  extremes  of  an  efficient  review  versus  a  thorough  review. 

Recent  test  activities  have  revealed  shortcomings  in  the  effectiveness  of  the  TRB 
process.  For  example,  it  was  found  that  one  particular  TRB  for  a  large  rocket  program 
concentrated  on  procedure  review  and  spent  very  little  time  evaluating  the  system  design. 
System  design  errors  could  have  lead  to  personnel  being  exposed  to  dangerous  chemicals. 
Even  though  this  is  the  most  extreme  consequence,  the  more  common  scenario  would  be  that 
the  design  errors  could  lead  to  test  failure  or  loss  of  test  data.  One  of  the  reasons  TRBs  are 
conducted  is  to  find  design  errors  before  they  impact  the  test  program. 

First,  a  problem  myself  and  others  who  participate  in  TRBs  have  identified  is  that 
they  are  not  an  efficient  use  of  participants  time.  This  is  due  partially  to  poor  meeting 
management.  The  other  part  of  this  problem  relates  to  the  current  practice  of  having  all 
topics  discussed  at  meetings  that  include  all  or  most  of  the  review  board  members,  even  if 
not  all  members  have  an  interest  or  expertise  in  the  topic  being  discussed.  Some  will  argue 
that  having  everyone  in  the  meeting  discussing  every  topic  leads  to  input  that  otherwise 
would  go  unheard.  This  is  true,  but  my  experience  has  shown  the  overwhelming  majority  of 
useful  comments  on  any  topic  being  discussed  come  from  those  with  expertise  in  that  topic. 
The  overall  effect  of  the  current  practice  is  to  cause  board  members  to  become  disengaged 
during  portions  of  the  reviews,  and  to  significantly  limit  the  amount  of  topics  a  TRB  can 
review  due  to  the  lack  of  ability  to  review  multiple  topics  simultaneously. 
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Second,  my  observations  of  the  TRB  process  have  revealed  that  too  much  time  is 
being  spent  during  TRBs  going  through  procedures  step  by  step,  while  not  spending  enough 
time  on  review  of  the  system  design  and  analysis.  Years  ago,  TRBs  started  placing  a  high 
level  of  emphasis  on  procedure  review  due  to  programs  generating  very  poor  quality 
procedures.  As  a  result,  review  of  system  design  and  analysis  became  secondary  and  on 
some  programs  not  even  a  part  of  the  TRB  review.  A  pattern  was  observed  where  programs 
with  experienced  test  directors  had  good  procedures  and  programs  while  those  with 
inexperienced  test  directors  had  poor  quality  procedures.  Recent  changes  in  AFRL  practice 
have  required  test  directors  to  be  experienced  in  rocket  testing.  Additionally,  last  November, 
a  local  procedure  writing  guidance  was  created  to  standardize  and  improve  the  quality  of 
procedures.  These  changes  should  allow  the  TRBs  to  reduce  the  amount  of  time  spent 
reviewing  procedures,  which  should  be  redirected  to  review  of  system  design  and  analysis. 

Significance  of  the  Topic 

The  majority  of  the  firsthand  knowledge  for  testing  rockets  during  the  glory  days  of 
the  rocket  industry  was  possessed  by  those  no  longer  part  of  the  workforce.  The  current 
rocket  testers  have  gained  experience  based  on  a  far  smaller  number  of  rocket  tests.  While 
the  current  generation  has  benefitted  from  the  experience  documented  by  those  that  have 
gone  before  them,  they  are  at  a  disadvantage  due  to  the  small  number  of  learning 
opportunities  resulting  from  the  significantly  reduced  number  of  rocket  tests.  The 
importance  of  a  good  quality  technical  review  has  never  been  more  important  in  order  to 
continue  successful  testing  and  protect  lives  due  to  the  limited  hands-on  experience  of 


current  rocket  testers. 
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Hands-on  experience  is  not  the  only  new  challenge  rocket  testers  face,  they  are  also 
performing  test  with  far  fewer  test  personnel.  The  number  of  test  personnel  has  decreased 
drastically  over  time  due  to  downsizing.  This  has  resulted  in  there  being  a  very  small  pool  of 
individuals  available  to  provide  technical  expertise  related  to  rocket  testing.  These 
individuals  are  busy  and  time  spent  on  one  program  usually  means  it  is  being  taken  away 
from  another  program.  There  is  rarely  any  depth  of  experienced  personnel  on  a  test  program 
anymore.  Now  more  than  ever  it  is  important  for  TRBs  to  be  managed  efficiently. 

A  paradigm  shift  in  the  way  technical  review  boards  are  conducted  at  AFRL  for 
rocket  test  programs  is  needed.  Despite  the  differences  between  rocket  test  programs,  there 
are  still  a  large  number  of  similarities  which  would  allow  for  detailed  local  TRB  guidance  to 
be  created.  Without  detailed  local  guidance,  TRBs  could  miss  important  technical  and  safety 
issues  with  a  program,  which  could  lead  to  test  failures  or  personnel  being  harmed. 

Stakeholders 

The  main  stakeholders  related  to  technical  review  boards  for  rocket  test  programs 
performed  by  the  Space  Missile  Propulsion  Division  are  the  following: 

•  AFRL  Management 

•  Technical  Review  Board  Members 

•  Test  Personnel 

•  Other  Government  Agencies  That  Test  Rockets 

•  Aerospace  Industry 

•  Taxpayers 

AFRL  management  creates  the  technical  review  boards  and  relies  on  the  review 
boards  to  provide  them  with  an  independent  assessment  of  the  programs  ability  to  meet  the 
technical  objectives.  These  reviews  must  consistently  provide  AFRL  management  with  high 
quality  feedback  for  the  process  to  work  effectively.  Poor  quality  technical  reviews  could 
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lead  to  problems  for  the  program  and  management,  such  as  personnel  being  injured  or 
equipment  being  destroyed. 

Technical  review  board  members,  including  the  technical  review  board  chairperson, 
invest  a  great  deal  of  time  and  effort  reviewing  programs.  Review  board  members  are 
selected  because  they  have  expertise  in  some  portion  of  testing,  similar  to  the  program  they 
are  reviewing  and  therefore  are  very  interested  in  the  quality  of  the  reviews.  Review  board 
assignment  is  a  duty  assigned  to  someone  in  addition  to  their  regular  responsibilities. 

Test  personnel  include  program  managers,  test  engineers,  technicians,  etc.  Test 
personnel  are  directly  responsible  for  program  success.  Test  engineers  design  and/or  review 
the  facility  design  and  analysis.  Test  personnel  write  the  procedures  and  perform  the  system 
checkouts.  Test  personnel  create  and/or  review  all  documents  which  are  given  to  the  review 
board.  Additionally,  as  discussed  in  the  previous  section,  technical  problems  can  turn  into 
safety  problems,  and  test  personnel  are  those  most  likely  to  be  affected  by  a  safety  problem. 

Other  government  agencies  like  the  Navy,  the  Army,  the  Missile  Defense  Agency, 
and  NASA  all  have  rocket  programs.  The  aerospace  industry  receives  funding  from 
government  agencies  to  build  and  test  rockets.  Many  of  the  tests  conducted  by  the  Space 
Missile  Propulsion  Division  are  for  rockets  funded  and/or  developed  by  one  of  these 
agencies.  Additionally,  a  failure  at  any  test  site  can  have  repercussions  for  the  entire  rocket 
test  community.  For  example,  a  catastrophic  failure  can  lead  to  new  safety  or  procedural 
requirements,  costly  upgrades  to  facilities,  or  schedule  delays  similar  to  what  happened  when 
the  Space  Shuttle  Columbia  blew  up. 

Most  rocket  test  programs  are  funded  by  the  government.  Successful  technical 
reviews  find  problems  before  they  happen  saving  time  and  resources.  When  programs  fail  it 
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usually  cost  the  taxpayers  by  diverting  funds  from  other  programs  or  abandoning  the  research 
the  program  that  was  being  worked. 

Overview  of  the  Thesis 

Chapter  2  of  this  thesis  is  the  literature  review.  The  literature  review  is  broken  into 
four  sections:  Complexity  in  organizational  decision  making;  Varieties  of  coping 
mechanisms;  Organizational  culture  change;  and  Meeting  management.  Complexity  in 
organizational  decision  making  and  varieties  of  coping  mechanisms  were  chosen  in  order  to 
understand  and  find  ways  to  deal  with  the  complex  decisions  technical  review  boards  must 
make.  The  topic  of  organizational  culture  change  was  studied  to  find  effective  methods  of 
implementing  changes  to  existing  processes.  Meeting  management  was  selected  to  find 
more  efficient  ways  to  conduct  TRB  meetings. 

Chapter  3  lists  the  methodology  for  this  thesis.  Chapter  3  begins  with  a  reflection  of 
my  personal  experience  related  to  the  TRBs.  It  is  then  followed  by  a  discussion  of  the 
literature  review  and  where  the  literature  was  obtained.  AFRL  is  a  secured  location  and 
requires  documentation  to  be  approved  for  release,  so  the  third  section  describes  the  AFRL 
public  release  process.  In  addition,  federal  law  requires  all  research  to  be  evaluated  to  make 
sure  human  research  subjects  are  treated  ethically  or  verified  that  no  human  subject  are 
involved.  The  chapter  finishes  with  a  section  on  the  CSUB  Institutional  Review  Board  (IRB) 
process  and  explains  how  this  study  was  not  to  be  human  subjects  research. 

Chapter  4  is  the  findings  and  contains  the  recommend  detailed  local  TRB  guidance 
for  rocket  test  programs  to  assist  TRB  chairperson  conduct  a  successful  review.  The 
guidance  includes: 
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1 .  A  standard  listing  of  items  to  be  reviewed  to  assist  TRB  chairperson  in  making 
sure  no  topic  is  missed. 

2.  How  to  plan  and  organize  a  TRB  in  order  to  perform  a  more  comprehensive 
review. 

3.  Suggestions  on  how  to  hold  efficient  meetings  and  perfonn  proper  resource 
management. 

Chapter  5  is  the  summary  and  conclusions.  This  chapter  also  contains  recommendations  for 
future  work. 
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CHAPTER  2 
Literature  Review 
Overview 

This  literature  review  is  broken  into  four  categories,  with  the  first  section  pertaining 
to  complexity  in  organizational  decision  making.  The  second  section  is  on  varieties  of 
coping  mechanisms,  and  explores  the  use  of  policies,  procedures,  and  checklists  as  a  way  to 
deal  with  the  complexity  in  organizational  decision  making.  The  third  section  is  on 
organizational  change,  and  it  examines  the  topic  of  quality  processes  including  Total  Quality 
Management  (TQM).  The  final  section  is  on  meeting  management,  and  was  included  to  find 
more  efficient  and  effective  ways  to  conduct  meetings.  The  majority  of  the  literature  was 
found  in  the  ProQuest  database,  which  is  a  large  research  database  available  to  CSUB 
students.  Other  sources  were  obtained  from  internet  searches,  professor  recommendations, 
reading  from  previous  class  work,  and  references  listed  in  other  articles. 

Complexity  in  Organizational  Decision  Making 

This  section  will  examine  topics  related  to  complexity  in  organizational  decision 
making.  The  literature  on  this  subject  includes  the  need  for  networking,  the  need  for 
guidance,  purpose  of  polices,  and  issues  associated  with  unwritten  policies. 

Our  volatile  and  dynamic  world  is  changing  at  an  amazing  rate  due  to  new 
technologies  and  the  availability  of  new  forms  of  instant  communication.  Skarzauskiene 
stated  (2010),  “The  environment  in  organizations  is  becoming  more  complex  and  changes 
more  often  and  suddenly  (Tvede,  1997;  Stacey,  1993;  Goswami,  1993;  Tetenbaum,  1998; 
Laszlo,  2002).”  (p.  50)  Weber  and  Khademian  relayed  (2008),  “As  early  as  1967,  scholars 
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and  practitioners  from  different  disciplines  recognized  that  the  dynamic  complexity  of  many 
public  problems  defies  the  confines  of  established  “stovepiped”  systems  of  problem 
definition,  administration,  and  resolution  (Churchman  1967;  Rittel  and  Webber  1973; 

Roberts  1997,  2000,  2002b).”  (p.  336)  Weber  and  Khademian’s  comments  were  in  the 
context  of  the  need  for  networking  between  different  government  agencies,  but  it  applies  to 
different  departments  within  an  organization.  Technical  review  boards  require 
multidiscipline  membership  to  help  prevent  stovepiping  and  the  guidance  for  conducting  a 
TRB  is  intended  to  establish  a  better  way  of  defining,  administering,  and  resolving  the 
problems  presented  to  the  TRB. 

As  seen  from  the  previous  paragraph,  stovepiping  can  be  an  organizational  concern, 
in  a  sense,  stovepiping  types  of  knowledge  can  be  just  as  big  of  a  problem.  Schmidt  relayed 
(1993),  “We  need  to  remember  that  all  kinds  and  sources  of  knowledge  are  not  superior  nor 
inferior  but  simply  different  ways  of  perceiving  and  organizing  our  limited  understandings  of 
a  rich  and  complex  reality.”  (p.  530)  Schmidt’s  reminder  that  our  complex  world  requires 
different  perspectives  is  very  important  to  the  success  of  the  technical  review  board  success. 
The  TRB  process  will  break  down  if  any  participant  in  the  TRB  process  starts  to  believe  their 
knowledge  or  sources  of  knowledge  are  superior  to  others. 

Giving  considerations  to  all  types  and  sources  of  knowledge  is  not  an  easy  task, 
therefore  tools  to  assist  in  the  processing  of  this  information  would  be  helpful.  Brown  found 
(2010),  “Public  Managers  typically  operate  in  highly  dynamic  and  complex  environments 
and  need  decision-making  heuristics  to  simplify  that  complexity.”  (p.  S213)  The  TRB 
guidance  created  by  this  study  is  designed  to  be  an  approach  that  helps  the  TRB  chairperson 
simplify  the  complex  TRB  process. 
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Schlarman  states  (2010),  “Policies  are  the  first  line  of  defense  against  risk  from  an 
organizational  perspective.”  (p.  1)  Schlarman  wrote  (2010),  “The  ultimate  goal  of  setting 
policies  is  to  influence  behavior,  set  clear  requirements  and  guide  people  through  business 
decisions.”  (p.  3)  While  the  Air  Force  does  have  regulations  for  TRBs  these  regulation  are 
too  vague  to  accomplish  the  goals  that  Schlarman  presented  for  setting  policies.  The  reason  I 
chose  this  topic  of  creating  guidance  for  the  TRB  chairperson  is  to  provide  the  level  of  detail 
needed  in  addition  to  the  Air  Force  regulation  to  accomplish  this  goal. 

Now  with  the  goal  in  mind  we  will  examine  the  current  state  of  the  organization  with 
regard  to  TRBs.  According  to  Van  Dusen  (2006),  if  you  are  a  small  organization  then 
policies  may  be  “understood”  even  if  not  in  writing.  The  Air  Force  Research  Laboratory  is 
not  a  small  organization,  and  for  large  organization  like  AFRL  “understood”  procedures 
can’t  possibly  work  as  there  is  no  way  to  make  sure  everyone  has  the  same  understanding. 
Van  Dusen  argued,  “Relying  on  ‘understood’  policies,  however,  may  lead  to 
misunderstandings.”  (p.  59)  It  is  my  opinion,  that  AFRL  works  to  “understood”  policies  on 
how  to  conduct  TRBs.  Many  AFRL  personnel  would  state  that  they  understand  the  purpose 
and  how  TRBs  are  to  be  perfonned  successfully.  Recent  experience  has  shown  that  there  is 
little  consistency  in  regards  to  AFRL  personnel’s  understanding  of  the  TRB  process.  This 
has  led  to  misunderstandings  and  inconsistencies  in  how  TRBs  are  performed,  evaluated,  and 
presented  to  management.  Van  Dusen  stated,  “Just  imagine  the  kind  of  resentment  and 
frustration  a  situation  like  this  might  create”  (p.  59).  This  situation  has  caused  arguing 
between  personnel  involved  in  TRBs.  Additionally  it  can  focus  the  reviewers  and  test  team 
on  the  arguments  instead  of  the  concentrating  on  what  needs  to  be  done.  Van  Dusen 
concluded,  “On  an  organization-wide  basis,  this  can  mean  lower  morale  and  productivity, 
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more  grievances  and  understandably  poor  relations  between  supervisors  and  employees”  (p. 
59).  If  you  substitute  supervisors  and  employees  for  review  board  members  and  test  team, 
this  statement  sums  what  sometimes  happens  at  TRBs. 

While  the  previous  paragraph  explains  the  issue  related  to  AFRL  working  with 
“understood”  policies,  it  is  important  to  note  that  the  belief  that  everyone  understands  the 
policies  will  lead  to  resistance  to  any  specific  guidance  being  accepted.  Semel,  Bader,  and 
Gawande  acknowledged  (2010)  “Successful  implementation  entails  local  system  change. 

But  there  is  evidence  that  such  efforts  can  produce  substantial  benefits.”  (p.  2355-2356)  The 
need  for  cultural  change  will  be  explored  more  in  the  section  on  organization  cultural  change. 

This  section  examined  several  concepts  related  to  complexity  in  organizational 
decision  making.  The  lessons  learned  from  this  section  include  the  importance  of 
multidiscipline  reviews,  being  receptive  to  different  forms  of  knowledge,  and  the 
significance  of  having  written  policies.  The  next  section  will  look  into  various  coping 
mechanisms  an  organization  can  use  to  handle  these  complexities  in  organizational  decision 
making. 

Variety  of  Coping  Mechanisms 

This  section  will  explore  ways  organizations  can  handle  the  complexities  that  are  part 
of  everyday  organizational  decision  making.  The  literature  on  this  subject  includes  why 
policies,  procedures,  and  checklists  are  needed  and  what  makes  a  good  checklist. 

From  How  to  Write  a  Policy  Manual,  “A  policy  is  a  predetennined  course  of  action 
established  as  a  guide  toward  accepted  objectives  and  strategies  of  the  organization”  (p.  5) 
According  to  Gawande  (2009),  “..The  volume  and  complexity  of  what  we  know  has 
exceeded  our  individual  ability  to  deliver  its  benefits  correctly,  safely,  or  reliably.”  (p.  13) 
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Van  Dusen  contended  (2006),  “Another  reason  for  developing  a  policy  manual  is  the 
increasing  difficulty  of  managing  and  controlling  complex  operations.”  (p.  62)  Van  Dusen 
wrote  (2006),  “A  policy  manual  is  an  excellent  training  resource.”  (p.  61)  AFRL  has  recently 
started  to  compile  a  Test  Operations  Handbook  in  part  to  be  used  as  a  training  resource.  This 
study  will  create  a  Test  Operations  Handbook  section  on  how  to  conducted  a  TRB  and 
include  a  checklist  of  possible  review  topics  to  assist  the  TRB  chairperson  in  managing  the 
complexities  involved. 

In  The  Checklist  Manifesto,  Atul  Gawande  explains  why  checklists  for  surgery  are 
needed  and  the  logic  is  applicable  to  a  technical  review  board  checklist.  Gawande  observed 
(2009), 

“Yet  it  is  far  from  obvious  that  something  as  simple  as  a  checklist  could  be  of 
substantial  help.  We  may  admit  that  errors  and  oversights  occur  -  even  devastating 
ones.  But  we  believe  our  jobs  are  too  complicated  to  reduce  to  a  checklist.”  (p.  34- 
35) 

Gawande  states  one  of  the  reasons  checklist  are  needed  (2009),  “The  first  is  the  fallibility  of 
human  memory  and  attention,  especially  when  it  comes  to  mundane,  routine  matters  that  are 
easily  overlooked  under  the  strain  of  more  pressing  events.”  (p.  36)  Gawande  further 
explains  (2009),  “. .  .people  can  lull  themselves  into  skipping  steps  even  when  they 
remember  them.  In  complex  processes,  after  all,  certain  steps  don’t  always  matter.”  (p.  36) 
Gawande  concludes  (2009),  “Checklists  seem  to  provide  protection  against  such  failures. 
They  remind  us  of  the  minimum  necessary  steps  and  make  them  explicit.  They  not  only 
offer  the  possibility  of  verification  but  also  instill  a  kind  of  discipline  of  higher 
performance.”  (p.  36)  Olson  and  Kelly  wrote  about  checklists  (2005),  “Expert  knowledge 


TECHNICAL  REVIEW  BOARD  CHAIRPERSON  GUIDELINES  FOR  20 

CONDUCTING  TECHNICAL  REVIEW  BOARDS  FOR  ROCKET  TESTING 

should  be  documented  because  experts  can  leave  your  organization,  taking  precious 
organizational  knowledge  with  them.”  (p.  60) 

I  found  Gawande’s  description  of  what  makes  a  bad  checklist  and  good  checklist  to 
be  accurate,  matching  my  personal  experience  with  checklists.  Gawande  explains  (2009), 
“Bad  checklists  are  vague  and  imprecise.  They  are  too  long;  they  are  hard  to  use; 
they  are  impractical.  They  are  made  by  desk  jockeys  with  no  awareness  of  the 
situations  in  which  they  are  to  be  deployed.  They  treat  people  using  the  tools  as 
dumb  and  try  to  spell  out  every  single  step.  They  turn  people’s  brains  off  rather  than 
turn  them  on. 

Good  checklist,  on  the  other  hand,  are  precise.  They  are  efficient,  to  the  point, 
and  easy  to  use  even  in  the  most  difficult  of  situations.  They  do  not  try  to  spell  out 
everything  -  a  checklist  cannot  fly  a  plane.  Instead,  they  provide  reminders  of  only 
the  most  critical  and  important  steps  -  the  ones  that  even  the  highly  skilled 
professionals  using  them  could  miss.  Good  checklist  are,  above  all,  practical.”  (p. 
120) 

When  creating  a  checklist  the  obvious  question  is  what  should  be  considering  in  the 
design  of  the  checklist?  Again  Gawande  provides  guidance  (2009),  “A  checklist  cannot  be 
lengthy.”  (p.123)  Gawande  reports  (2009),  “People  start  “shortcutting”.  Steps  get  missed. 
So  you  keep  the  list  short  by  focusing  on  what  he  called  “the  killer  items”  -  the  steps  that  are 
most  dangerous  to  skip  and  sometimes  overlooked  nonetheless.”  (p.  123)  Gawande  states 
(2009),  “The  wording  should  be  simple  and  exact. ...”  (p.  123) 

In  the  previous  paragraph  the  discussion  on  checklist  centered  around  what  elements 
make  a  good  checklist.  The  lessons  in  this  paragraph  are  related  to  the  purpose  of  the 
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technical  review  board  and  one  of  the  key  items  the  review  board  evaluated.  Part  of  the 
AFRL  test  process  requires  test  programs  create  a  verification  and  validation  plan  (V&V 
plan).  The  V&V  plan  lists  the  inspections,  analysis,  and  functional  checkouts  that  will  be 
performed  to  provide  confidence  the  test  will  be  successful.  While  not  in  the  standard 
checklist  format,  the  V&V  plan  is  essentially  a  checklist  designed  to  assist  the  test  team  in 
finding  defects  before  they  affect  the  test.  From  the  field  of  software  engineering  comes  an 
interesting  perspective  on  checklists.  In  the  article  A  Statistical  Approach  to  the  Inspection 
Checklist  Formal  Synthesis  and  Improvement  the  Chernak  presents  the  elements  needed  in  an 
inspection  checklist  to  detect  flaws  in  software.  Chemak  found  (1996), 

“However,  according  to  our  experience,  an  inspection  process  becomes  more 
productive  and  the  inspectors  find  more  defects  when  a  checklist  is  represented  by 
two  components.  The  first  component  is  of  a  “Where  to  look”  category  and  means  a 
particular  feature  of  a  work  product.  It  should  lead  the  inspector  to  a  specific  “spot” 
in  the  work  product  where  a  given  rule  is  implemented.  The  second  component  is  of 
a  “How  to  detect”  category.”  (p.  867) 

Chernak  stated  (1996),  “Hence,  a  defect  model  should  have  one  more  attribute  -  type  of 
inspection.”  (p.  868)  Basically  the  V&V  plan  should  be  designed  to  find  “defects”  that 
would  lead  to  failure.  One  of  the  main  tasks  of  the  TRB  is  to  evaluate  whether  the  test 
team’s  following  the  V&V  plan  will  lead  to  a  high  level  of  confidence  of  technical  success. 
The  TRB  guidance  should  include  verifying  that  the  V&V  plan  includes  information  on 
where  the  requirements  come  from,  how  to  detect  compliance,  and  what  type  of  inspection, 
analysis,  checkout  or  combination  of  these  will  be  used  to  verify  the  systems  are  ready  for 


test. 
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This  section  explored  coping  mechanisms  for  dealing  with  the  complexities 
organizations  face.  The  lessons  learned  from  this  section  included  reasons  for  having 
policies,  elements  of  good  and  bad  checklists,  and  advice  that  can  be  used  for  the  review  of 
the  V&V  plan.  The  next  section  will  examine  organizational  culture  change  and  how  lessons 
from  the  quality  assurance  community,  including  how  to  get  employee  buy-in,  can  shed  light 
into  effective  ways  to  bring  about  change  in  an  organization. 

Organizational  Culture  Change 

This  section  will  look  into  the  topic  of  organizational  culture  change.  The  literature 
on  this  subject  includes  a  review  of  quality  process  with  an  emphasis  on  total  quality 
management  (TQM).  Total  Quality  Management  (TQM)  is  a  quality  assurance  philosophy 
based  on  W.  Edwards  Deming  14  points.  Deming’s  teachings  are  credited  for  much  of  the 
quality  improvements  in  the  Japanese  auto  industry. 

There  is  a  lot  of  debate  if  TQM  as  a  whole  can  be  applied  to  the  public  sector,  but 
there  is  little  doubt  that  parts  of  TQM  can  be  applied  to  the  public  sector.  Swiss  reported 
(1992), 

“TQM  has  been  endorsed  by  President  Bush,  who  said,  ‘Reasserting  our  leadership 
will  require  a  firm  commitment  to  total  quality  management  and  the  principles  of 
continuous  improvement. . . .  Quality  improvement  principles  apply. .  .to  the  public 
sector  as  well  as  private  enterprise’  (Car  and  Littman,  1990,  p.2)”  (p.  356) 

For  the  purposes  of  this  paper  the  debate  of  applying  TQM  as  a  whole  to  the  public  sector 
will  be  ignored  and  only  the  portions  of  TQM  and  other  quality  principles  that  can  be  applied 
directly  to  the  TRB  process  will  be  considered. 
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Swiss  found  (1992),  “...quality  requires  total  organizational  commitment.”  (p.  358) 
The  current  management  environment  of  AFRL  is  committed  to  quality  and  continuous 
process  improvement.  The  AFRL  has  joined  the  OSHA  Voluntary  Protection  Program 
(VPP),  which  parallels  the  TQM  process  but  with  an  emphasis  on  safety  instead  of  quality. 
Tompkins  warns  (2005),  “Without  a  fundamental  change  in  culture,  longstanding  priorities 
and  behaviors  will  persist.”  (p.  345)  The  principles  of  the  VPP  program  include  employee 
engagement  and  allowing  those  at  all  levels  of  the  organization  to  play  an  active  part  in 
creating  a  safe  environment  with  the  support  of  management.  AFRL  has  been  in  the  process 
of  cultural  change  as  part  of  the  VPP  program  where  employees  are  starting  to  see  they  can 
personally  make  a  difference  and  are  receiving  management  support,  and  the  success  there 
will  easily  transfer  to  quality  process. 

According  to  Swiss  (1992),  “  ...quality  improvement  requires  strong  worker 
participation.”  (p.  358)  Tompkins  wrote  about  one  of  Deming’s  14  points  (2005), 

“Remove  barriers  to  pride  in  workmanship.  People  want  to  take  pride  in  their 
work.  It  is  a  primary  source  of  motivation.  Yet  little  pride  can  be  taken  where 
workers  are  subject  to  rules  and  policies  that  make  no  sense,  reward  systems  that  are 
geared  to  quantity  rather  than  quality,  and  no  opportunities  to  help  improve  the 
system.”  (p.  341) 

One  of  my  many  criticism  of  TRBs  from  the  test  team  perspective  has  been  that  the  test  team 
is  often  unable  to  take  pride  in  their  work  due  to  the  review  board  making  review  board 
member’s  preferences  requirements.  This  has  often  caused  test  personnel  to  put  minimal 
effort  into  documents  that  will  be  reviewed  by  the  boards.  They  believe  that  the  review 
board  will  take  ownership  of  anything  created,  therefore  making  the  effort  a  waste  of  time. 
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The  current  VPP  environment  has  given  new  hope  that  test  personnel  will  have  an 
opportunity  to  help  improve  this  process,  because  of  the  emphasis  the  VPP  program  puts  on 
having  management  support  those  closest  to  the  work  getting  the  support  required  to  make 
improvements.  Tribus  quoted  Deming  as  having  said  (1996),  “Inspection  does  not  build 
quality,  the  quality  is  already  made  before  you  inspect  it.  It’s  far  better  to  make  it  right  in  the 
first  place.”  (p.  10)  The  TRB  guidance  created  by  this  study  should  include  a  separation  of 
the  roles  and  responsibilities  to  clearly  define  who,  test  team  versus  review  board,  is 
responsible  for  what.  By  empowering  the  test  team,  they  will  be  more  motivated  to  create 
better  quality  products  in  the  first  place. 

All  the  discussion  in  the  previous  paragraph  about  employee  participation, 
motivation,  and  empowerment  applies  equally  to  review  board  members  as  it  does  to  test 
team  members.  My  experience  and  observation  of  others  who  have  participated  as  review 
board  members  shows  that  review  board  members  want  to  contribute  to  the  success  of  a 
program.  Review  board  members  are  participating  in  addition  to  their  normal  workload  and 
therefore  would  like  their  time  spent  efficiently  during  the  review  process.  The  next  section 
will  discuss  meeting  management,  which  is  one  way  to  efficiently  use  review  board  members 
time.  From  lessons  learned  by  NASA,  Harkins  stated  (1999), 

“As  a  precursor  to  the  design  review,  detailed  technical  reviews  (DTRs)  are 
recommended  for  important  products  to  facilitate  early  detection  and  correction  of 
design  deficiencies.  DTRs  are  infonnal,  working-level,  peer  reviews. .  .The  primary 
differences  are  the  greater  technical  depth  provided  by  the  DTR,  the  corresponding 
higher  specialization  of  the  small  (commonly  2  to  3-person)  review  team...”  (Review 
Procedures  section,  para.  2) 
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The  design  review  presented  by  Harkins  is  similar  to  a  TRB.  DTRs  do  not  officially  exist  for 
programs  within  the  Space  Missile  Propulsion  Division.  Many  programs  perform  unofficial 
peer  reviews,  like  the  DTRs,  by  having  several  test  team  members  review  and  comment  prior 
to  material  being  submitting  to  the  review  board.  These  programs  in  my  experience  have 
generally  had  shorter  TRB  reviews  due  to  better  quality  starting  documentation.  Based  on 
the  NASA  lessons  learned  and  personal  experience,  I  would  recommend  requiring 
documentation  to  be  submitted  to  review  boards  with  a  peer  review  page  signed  by  at  least 
two  test  team  members  other  than  the  author  of  the  document. 

As  discussed  in  the  previous  paragraphs,  a  test  team  creating  poor  quality 
documentation  is  a  problem  but  just  as  big  of  a  problem  is  the  acceptance  of  poor  quality 
materials  that  are  submitted  for  review.  Swiss  reported  (1992),  “Accordingly,  TQM 
generally  opposes  mass  inspection  of  products  because  such  inspections  provide  a  safety  net 
that  shifts  quality  responsibilities  away  from  the  initial  designers  and  producers.”  (p.  357) 
Sometimes  the  poor  quality  initial  documentation  being  submitted  to  the  review  boards  is 
more  related  to  the  safety  net  issue  Swiss  presents.  If  the  review  boards  are  going  to  spend 
days  or  months  reviewing  the  test  team’s  documentation,  regardless  of  its  initial  quality, 
there  is  little  incentive  to  put  a  lot  of  effort  into  the  draft  submitted.  The  first 
recommendation  I  have  from  reading  Swiss  statement  is  for  TRB  chairperson  not  to  accept 
poor  quality  documents  for  review.  Instead  the  chairperson  should  send  the  document  back 
to  the  test  team  for  improvement.  If  the  test  team  continually  produces  poor  quality 
documentation,  then  the  TRB  chairperson  should  request  the  AFRL  West  Test  Lead  and/or 
responsible  branch  chief  to  provide  the  test  program  with  someone  with  the  required 
expertise  to  assist.  The  second  recommendation  is  that  the  TRB  not  review  all 
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documentation  in  detail.  Each  document  should  only  receive  the  amount  of  review  that  is 
required,  unless  the  review  finds  major  flaws. 

This  section  has  explored  the  problems  of  quality  documentation,  now  the  focus  shifts 
specifically  to  the  documentation  of  requirements.  Requirements  for  rocket  test  programs 
come  from  many  sources  to  include,  but  not  limited  to,  the  test  plan,  regulations,  policies, 
and  best  practices.  According  to  Deming  (1952-1953),  “’Good  quality’  and  ‘uniform 
quality’  have  no  meaning  except  with  reference  to  the  customer’s  needs.”  (p.  430)  The 
customer’s  needs  are  documented  in  the  test  plan.  The  TRB’s  job  is  to  verify  that  the  test 
plan  requirements  can  be  met.  Hoyer  and  Hoyer  quoted  Phillip  Crosby  (2001),  “That  is 
precisely  the  reason  we  must  define  quality  as  ‘conformance  to  requirements’  if  we  are  to 
manage  it.”  (p.  54)  Hoyer  and  Hoyer  stated  (2001),  “Somehow,  someone  must  know  what 
the  requirements  are  and  be  able  to  translate  those  requirements  into  measurable  product  or 
service  characteristics.”  (p.  55)  Requirements  no  matter  the  source,  test  plan,  regulation,  best 
practice,  etc.,  are  translated  into  the  V&V  plan. 

As  previously  stated,  one  of  the  primary  functions  of  the  TRB  is  to  evaluate  the 
effectiveness  of  the  verification  and  validation  plan  (V&V  plan).  The  V&V  plan  is  the  test 
teams  approach  to  ensuring  technical  success  of  the  program.  The  V&V  plan  tells  what 
inspection,  analysis,  and  functional  checkouts  will  be  performed.  Deming  wrote  (1947), 

“The  problem  of  validity  is  one  of  the  proper  approach  to  the  problem,  to  discover  what 
information  is  wanted  and  to  find  out  how  to  draw  it  out  if  it  can  be  had.”  (p.  147)  Deming 
concludes  (1947),  “In  short,  it  is  the  problem  of  eliciting  the  right  answers  to  the  right 
questions.”  (p.  147)  The  TRB  is  required  to  evaluate  whether  following  the  V&V  plan  will 
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provide  a  high  likelihood  of  success.  Basically,  is  the  approach  proper  to  the  problem  and 
will  that  result  in  the  right  answers? 

This  section  looked  into  organizational  culture  change  by  reviewing  the  literature  on 
quality  processes.  The  lessons  learned  from  this  section  included  the  need  for  test  team  and 
review  board  member  roles  and  responsibilities,  peer  reviews,  and  the  end  of  the  acceptance 
of  poor  quality  documentation.  This  section  finished  by  focusing  on  the  V&V  Plan.  The 
next  section  will  explore  the  topic  of  meeting  management. 

Meeting  Management 

TRB  meetings  at  AFRL  tend  to  be  long,  drawn  out  meetings,  often  taking  what 
should  be  completed  in  less  than  two  hours  and  stretching  it  into  half  day  or  full  day  events. 
Lee  found  (2008),  “Just  as  good  project  management  makes  for  a  successful  project,  proper 
management  of  meetings  makes  them  more  successful.”  (p.  25)  Landale  wrote  in  regards  to 
meetings  (2004),  “They  have  to  deal  with  the  essentials,  fast  and  effectively,  and  this  requires 
a  completely  different  focus  for  both  the  person  facilitating  the  meeting  and  from  attending 
participants.”  (p.26)  Landale  quoted  Rob  Cram,  STC’s  managing  director,  as  having  said 
(2004), 

“Getting  radical  with  meetings  means  ending  this  complacent  practice  and,  instead, 
working  with  action-oriented  objectives.  This  approach  focuses  on  the  business 
priorities  that  the  team  needs  to  resolve  and  requires  that  every  participant  come  to 
the  meeting  with  ‘thought  through’  ideas  on  how  this  can  be  achieved.”  (p.  26) 

The  first  lesson  on  meeting  management  taken  from  the  literature  is  to  make  meetings 
focused  with  action-oriented  objectives. 
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Allen,  Rogelberg,  and  Scott  pose  several  questions  that  should  be  asked  about 
meetings,  the  first  of  which  is  (2008),  “Can  topics  of  meetings  be  handled  through  e-mails?” 
(p.  50)  A  lot  of  topics  that  are  part  of  the  TRB  process  can  be  handled  through  e-mail,  so  the 
lesson  here  is  “don’t  meet”.  Allen  et  al.  also  asked  (2008),  “Do  real  outcomes  and  action 
items  result  from  the  gathering?”  (p.  50)  Often  TRBs  become  regularly  scheduled  meetings 
until  completed,  but  if  there  is  no  expected  outcome  or  action  items  from  any  particular 
meeting  then  that  meeting  should  be  cancelled.  Allen  et  al.  further  asked  (2008),  “Was  any 
assigned  pre-work  completed?  Were  action  items  accomplished  from  a  previous  meeting?” 
(p.  50)  If  the  required  documents  have  not  been  completed  prior  to  the  TRB  or  the  action 
items  from  the  previous  meeting  have  not  been  completed,  then  don’t  meet.  Allen, 
Rogelberg,  and  Scott’s  advice  seems  to  be  common  sense  and  obvious,  but  stopping  to 
answer  these  simple  questions  could  save  a  lot  of  time. 

All  the  authors  that  discussed  meeting  management  wrote  about  the  importance  of 
agendas.  Kloppenborg  and  Petrick  contended  (1999),  “The  agenda  should  include  logistics 
issues  (date,  time,  and  place),  the  meeting  purpose,  announcements,  decisions,  discussions, 
summary,  path  forward,  and  evaluation  (+/A)  sections.”  (p.  172)  Lee  reported  (2008),  “Write 
the  agenda  with  action  words  stating  what  the  participants  will  do  at  the  meeting.”  (p.  27) 
Allen  et  al.  agrued  (2008),  “The  agenda  should  start  with  the  most  critical  and  strategic 
items.”  (p.  52)  Staren  wrote  (2009), 

“It  is  important  to  specify  an  agenda  for  each  meeting  so  that  participants  can  plan  and 
prepare  appropriately  prior  to  getting  together.  The  chair  should  request  input  from  all 
members  regarding  agenda  topics  and  should  do  so  with  adequate  time  prior  to  the 
meeting.”  (p.  82) 
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According  to  Kloppenborg  and  Petrick  (1999),  “Individual  participant  or  committee  reports 
can  be  attached  to  the  agenda  with  progress  reports  on  projects  that  are  publicly  anticipated.” 
(p.  172)  In  the  case  of  TRBs  these  reports  will  often  be  the  document  the  test  team  has 
created  for  review  or  prep  work  performed  outside  the  meeting. 

Lee  maintained  (2008),  “Work  outside  the  meeting  room  gets  accomplished  to  carry 
out  the  meeting’s  purpose.”  (p.  28)  Current  Space  Missile  Propulsion  Division  practice  for 
TRBs  is  for  documents  to  be  reviewed  outside  the  meeting  and  for  the  members  to  come 
together  to  discuss  everyone’s  comments.  A  large  number  of  the  comments  don’t  require 
any  discussion  and  it  would  save  meeting  time  if  all  document  reviews  were  given  to  the  test 
team  to  incorporate  outside  the  meeting,  saving  time  in  the  meeting  for  only  topics  that 
required  discussion.  Landale  argued  (2004),  “It  requires  that  every  participant  in  every 
meeting  comes  ready  to  add  value.  It  also  requires  people  to  have  thought  about,  and  arrive 
ready  to  speak  about,  the  business  objectives  under  discussion.”  (p.  26)  Due  to  busy  work 
schedules  it  is  not  uncommon  for  some  TRB  participates  to  come  to  the  meetings 
unprepared.  This  practice  is  wasteful  and  should  be  discouraged.  Allen  et  al.  stated  (2008), 
“Provide  an  opt  out  clause.  Organizations  should  make  it  OK  for  employees  to  opt  out 
of  meeting  that  have  little  to  do  with  their  tasks  or  expertise,  imagine  the  cost  savings  of 
meeting  with  only  the  people  who  need  to  know,  as  opposed  to  the  whole  group  or  team.” 
(p.  52) 

Review  board  members  being  unprepared  for  the  meeting  can  also  result  from  the  meeting 
having  little  to  do  with  their  expertise  and  Allen,  Rogelberg,  and  Scott’s  opt  out  clause  could 
save  time.  The  current  Space  Missile  Propulsion  TRB  practice  is  to  meet  with  as  whole  a 
group  as  possible  for  every  topic  discussed.  Meeting  with  only  the  people  who  know  would 
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be  of  great  valued  to  board  and  test  team  members  by  eliminated  the  need  to  get  the 
unprepared  review  board  member  up  to  speed.  Furthennore,  by  allowing  an  ‘opt  out’  rather 
than  ‘not  inviting’  means  that  the  team  members  lack  of  attendance  is  acknowledgement  of 
agreeing  to  the  outcome  by  the  present  TRB  members. 

Going  back  to  some  of  the  principles  presented  earlier  in  this  section,  meetings  should 
result  in  decisions  and  action  items.  Lee  and  Lazarus  wrote  (1993),  “Agree  on  follow-up 
actions.  This  is  done  by  clearly  defining  what  they  are,  who  will  be  responsible  for  them, 
and  when  they  will  be  completed.”  (p.  1 1)  According  to  Lee  (2008),  “Assigning  actions 
gives  members  the  opportunity  to  contribute  at  an  individual  level  and  ensures  personal 
responsibility  and  accountability  for  achieving  the  overall  purpose.  Having  due  date  lets  the 
members  plan  their  time  outside  the  meeting  for  personal  productivity.”  (p.  28)  TRBs  should 
assign  more  individual  responsibilities  to  the  board  members  by  assigning  action  items  to  be 
performed  outside  of  the  meeting  allowing  all  board  members  to  contribute  to  the  process. 
Staren  contended  (2009),  “Subsequent  to  the  meeting  it  is  essential  that  the  group  is  sent 
accurate,  organized  minutes  in  a  timely  manner.”  (p.  83)  These  meeting  minutes  should 
include  the  action  items,  who  is  responsible,  and  the  due  dates. 

Allen  et  al.  give  us  this  warning  about  trying  to  change  meeting  behavior  (2008), 
“Meeting  behaviors  and  practices  are  often  well-learned,  institutionalized  habits  that  can  be 
difficult  to  break.”  (p.  52) 

This  section  examined  the  topic  of  meeting  management.  Lesson  learned  from  the 
literature  included  having  action-oriented  objectives,  when  to  meet,  and  the  importance  of 
agendas  and  meeting  minutes. 
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The  literature  review  looked  at  the  topics  of  complexity  in  organizational  decision 
making,  variety  of  coping  mechanisms,  organizational  culture  change,  and  meeting 
management.  The  complexity  in  organizational  decision  making  stated  the  criticality  of 
guidance  as  created  as  a  result  of  this  study.  The  variety  of  coping  mechanisms, 
organizational  culture  change,  and  meeting  management  sections  provided  advice  on  specific 
items  to  be  included  in  the  guidance. 
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CHAPTER  3 
Methodology 

Experience  and  Best  Practices 

A  significant  portion  of  the  guidance  created  will  be  based  on  personal  experience, 
but  backed  by  examples  from  literature  and  sound  systems  engineering.  I  have  been 
involved  with  the  TRB  process,  and  its  predecessor  the  technical  evaluation  committee,  for 
fourteen  years.  I  have  served  as  both  a  TRB  member  and  TRB  chairperson.  In  addition,  my 
experience  on  the  program  side  has  been  as  test  director,  test  conductor,  or  program  manager 
of  programs  going  through  a  technical  review.  Additionally,  I  have  recently  served  as  the 
lead  for  an  internal  technical  investigation  which  offered  a  unique  perspective  for  reviewing 
TRBs  after  they  have  been  completed.  The  experience  with  the  internal  technical 
investigation  lead  to  insights  and  conclusion  about  the  TRB  process  that  otherwise  I  would 
not  have  had.  The  investigation  revealed  to  me  that  missing  one  design  error  could  result  in 
personnel  being  exposed  to  dangerous  chemicals,  luckily  it  did  not  happen  in  this  case.  This 
was  despite  the  months  of  review  that  were  spent  examining  the  procedure  line  by  line.  Also 
the  investigation,  showed  me  how  different  review  boards  focus  on  different  topic  and 
allowed  me  to  reflect  on  how  few  perform  a  thorough  design  review  unless  prompted  by  the 
test  team  or  management. 

Over  the  years  I  have  been  involved  with  numerous  discussions  with  AFRL  personnel 
on  the  topic  of  TRBs  and  how  to  improve  them.  I  have  not  taken  any  actions  up  until 
recently,  when  AFRL  management  tasked  me  with  reviewing  and  creating  local  guidance 
related  on  the  subject  of  rocket  testing.  Most  of  my  initial  interest  was  related  to  the  process, 
knowledge  of  the  process,  tasking  to  create  local  guidance,  and  need  to  write  a  thesis  for  my 
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masters.  My  goal  is  to  improve  and  standardize  the  Space  Missile  Propulsion  Division’s 
approach  to  technical  review  boards  in  order  to  prevent  test  failures  and  personnel  injury. 

Best  practices  will  be  incorporated  into  the  guidance  created,  since  most  of  the  best 
practices  related  to  TRBs  have  not  been  written  down.  They  would  fit  what  Van  Dusen  calls 
“understood”  polices.  Faults  with  this  verbal  system  include  the  fact  that  infonnation  is 
passed  on  through  mentoring  and  word  of  mouth,  meaning  no  two  employees  receives  the 
exact  same  information  on  best  practices.  Additionally,  not  all  groups  within  AFRL  are 
equally  aware  of  the  existing  best  practices.  When  these  best  practices  are  used  they  have 
contributed  to  the  success  of  AFRL’s  rocket  test  programs.  A  major  reason  for  my  tasking  in 
the  creation  of  the  Test  Operations  Handbook  was  to  document  these  best  practices  for  the 
purpose  of  standardization  and  training  to  continue  and  possibly  improve  test  program 
success. 

Literature  Review 

A  literature  review  was  perfonned  as  part  of  this  study,  as  was  discussed  in  Chapter 
2.  The  literature  review  consisted  of  five  major  categories:  Complexity  in  organizational 
decision  making;  Varieties  of  coping  mechanisms;  Organizational  culture  change;  Meeting 
management;  and  Technical  Reviews. 

Most  of  the  literature  used  for  this  paper  came  from  the  on-line  version  of  the 
California  State  University,  Bakersfield  (CSUB)  Walter  Stiern  Library.  The  searches  were 
done  in  the  ProQuest  database  with  the  boxes  checked  for  full  text  documents  only  and 
scholarly  journals,  including  peer-reviewed.  Searches  were  performed  using  the  following 
keywords:  policies,  procedures,  checklist,  Deming  (both  as  author  and  document  text),  Total 
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Quality  Management,  TQM,  meeting  management,  organizational  decision  making, 
complexity,  and  technical  review. 

Additional  literature  was  found  using  internet  search  engines,  readings  from  previous 
class  assignments,  and  using  the  references  listed  in  other  articles.  The  book  by  Atul 
Gawande  The  Checklist  Manifesto:  How  to  get  Things  Right  was  recommended  by  Dr. 

Moore  and  Dr.  Commuri.  The  pertinent  Air  Force  regulations  were  obtained  from  the  Air 
Force  Research  Laboratory. 

The  search  for  information  on  complexity  in  organizational  decision  making  resulted 
in  a  wide  variety  of  different  source  with  the  intent  of  getting  a  general  overview  of  the  topic. 
Articles  that  went  into  details  on  narrow  topics  were  not  used.  Therefore,  a  small  sampling 
of  the  literature  was  used  due  to  the  common  theme  that  organizational  decision  making  is 
complex  and  the  lack  of  need  for  a  thorough  exploration  of  this  specific  topic  as  a  part  of  this 
study. 

The  varieties  of  coping  mechanisms  and  the  organizational  culture  change  sections  of 
the  literature  search  yielded  an  abundance  of  sources.  In  order  to  not  dilute  the  overall  focus 
of  this  thesis,  a  sampling  of  the  information  on  those  topics  was  used. 

The  meeting  management  section  of  the  literature  review  also  resulted  in  a  large 
number  of  available  references.  Abstracts  or  introductions  were  skimmed  for  key  words  to 
narrow  the  base  of  articles  to  those  relevant  to  the  topic.  Unlike  the  first  two  sections  of  the 
literature  review,  the  review  of  literature  on  meeting  management  was  wrapped  up  when  the 
articles  started  to  present  redundant  information. 

The  topic  of  technical  reviews  was  not  included  in  chapter  2.  The  literature  reviewed 
on  technical  reviews  appeared  to  be  more  related  to  a  large,  high-level  technical  review 
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meetings  or  very  specific  to  the  software  industry.  Most  of  this  information  did  not  apply  to 
the  technical  review  board  process  at  AFRL.  A  fifth  section  of  technical  review  was  not 
included  in  Chapter  2,  because  the  information  on  technical  reviews  that  did  apply  to  the 
AFRL  TRB  process  was  easily  fit  into  one  of  the  other  four  categories. 

AFRL  Public  Release  Process 

The  tie-in  to  work  for  the  Air  Force  for  this  study  necessitated  its  approval  by  the  Air 
Force  for  public  release.  A  position  paper  for  master’s  thesis  on  technical  review  board 
chairman  guidelines  for  conducting  technical  review  boards  for  rocket  testing  was  submitted 
into  the  process  for  public  release.  This  paper  was  approved  for  public  release  and  given  PA 
#11219  and  allowed  this  study  to  proceed  in  greater  detail  for  the  master’s  thesis  topic. 

There  is  an  Air  Force  form  that  must  be  filled  out  and  submitted  for  approval  for  any 
public  release  document.  The  creator  of  the  document  fills  out  the  form,  which  contains 
several  questions  related  to  document  classification.  Then  the  program  manager  (or  in  this 
case  the  student)  and  the  supervisor  review  the  document  and  sign  the  form.  The  form  is 
submitted  to  the  scientific  and  technical  information  office.  The  scientific  and  technical 
information  office,  technical  advisor,  foreign  disclosure  office,  and  public  affairs  office  all 
review  and  approve  the  document  for  public  release.  The  rocket  science  and  technology 
developed  by  AFRL  is  critical  for  maintaining  Air  Force  superiority,  and  the  public  release 
process  was  established  to  prevent  infonnation  from  getting  into  foreign  adversaries’  hands. 

This  master’s  thesis  was  approved  through  the  same  process  and  has  been  give 


PA# 
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CSUB  Institutional  Review  Board  (IRB)  Process 
All  students  in  the  master’s  program  are  required  to  complete  the  CSUB  IRB:  Human 
Subjects  Protection  Training.  The  training  was  completed  on  March  1 1,  201 1. 

No  interviews,  focus  groups,  or  surveys  were  performed  as  part  of  this  study.  All 
references  to  discussions  with  others  are  from  past  experience  prior  to  starting  the  study  and 
have  not  been  recorded  or  documented  anywhere.  No  reference  was  made  to  any  individual, 
only  reference  to  job  titles  or  generic  group  descriptions  were  used  in  this  study. 

Additionally,  the  Air  Force  reviewed  and  approved  this  thesis  for  public  release. 

Master’s  thesis  projects  are  required  to  be  reviewed  by  the  California  State 
University,  Bakersfield  Institutional  Review  Board  to  ensure  all  appropriate  human  subject 
safety  measures  are  applied.  The  form  titled  “Is  My  Project  Human  Subjects  Research?’  was 
filled  out  and  submitted  to  Dr.  Commuri  the  faculty  research  mentor.  He  obtained  the 
department  chair’s  signature  and  submitted  the  fonn  to  the  IRB.  This  study  was  reviewed 
and  detennined  not  to  be  human  subjects  research  (appendix). 


TECHNICAL  REVIEW  BOARD  CHAIRPERSON  GUIDELINES  FOR 
CONDUCTING  TECHNICAL  REVIEW  BOARDS  FOR  ROCKET  TESTING 


37 


CHAPTER  4 
Findings 

Conducting  a  Technical  Review  Board  Guidance 
This  study  has  resulted  in  the  creation  of  guidance  for  conducting  a  technical  review 
board  for  rocket  test  programs  perfonned  by  the  Space  Missile  Propulsion  Division  that  will 
be  helpful  to  TRB  chairpersons  and  useful  information  for  TRB  members.  The  guidance  is 
written  specifically  for  the  TRB  chairperson  as  he  is  responsible  for  organizing  TRB 
activities.  The  guidance  created  for  conducting  a  technical  review  board  is  in  appendix  B. 
The  guidance  is  in  the  Test  Operations  Handbook  format  used  by  the  Space  Missile 
Propulsion  Division.  The  remainder  of  this  chapter  will  explain  the  guidance  and  the  criteria 
used  when  creating  it. 

Header.  Footer.  Format,  Sections  I  and  II 
This  section  explains  the  standard  format  used  for  test  operation  handbook.  The 
header  contains  a  date  in  the  upper  left  comer.  This  date  is  used  to  track  the  date  of  the  draft 
prior  to  approval  and  is  then  replaced  with  the  approval  date  once  adopted.  In  the  center  of 
the  header  is  the  Test  Operations  Handbook  number  and  revision.  The  handbook  documents 
are  given  a  number  to  allow  them  to  be  put  in  the  procedure  management  system  and  take 
full  advantage  of  the  available  database  search  options.  The  number  starts  with  TOHB 
which  stands  for  Test  Operations  Handbook.  The  middle  portion  of  the  number  designates 
the  test  area  the  handbook  guidance  applies  to  or,  in  this  case,  AFRL  as  it  is  intended  for  use 
throughout  the  lab.  The  end  of  the  number  is  a  unique  four  digit  number  assigned  to  the 
handbook  guidance  once  adopted,  currently  XXXX  has  been  put  in  as  a  placeholder.  The 
revision  number  follows  the  TOHB  number  in  the  form  of  Rev.  X  and  for  an  initial  release  X 
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is  0.  In  the  right  hand  comer  of  the  header  is  the  page  number  and  total  number  of  pages  in 
the  form  of  Page  X  of  Y.  In  the  bottom  right  corner  of  the  footer  is  the  title  of  the  handbook 
guidance,  which  is  placed  there  to  assist  looking  through  the  handbook  when  multiple  topics 
have  been  put  together. 

The  Test  Operations  Handbook  guidance  has  two  columns  for  the  discussion  portion 
and  then  examples  and  attachments  are  at  the  end  in  whatever  format  is  appropriate  for  that 
handbook  guidance.  In  the  upper  left  corner  of  the  first  column  is  the  title  of  the  handbook 
guidance,  this  section  is  titled  Conducting  a  Technical  Review  Board.  The  first  section  is 
always  the  OPR,  which  stands  for  office  of  primary  responsibility.  The  second  section  is 
always  general  information  and  includes  the  purpose.  The  second  section  also  includes 
information  about  the  specific  guidance  is  applicable  to  what  test  activities.  After  the  second 
section  the  handbook,  sections  are  no  longer  standardize  from  one  handbook  section  to  the 
next,  but  instead  are  all  broken  down  into  parts  specific  to  the  topic  of  that  handbook  section. 

Section  III  Prior  to  the  First  Technical  Review  Board  Meeting 

The  purpose  of  section  III  of  the  guidance  was  to  help  the  TRB  chairperson  in 
planning  the  TRB  review  for  that  program.  The  first  paragraph  of  this  section  takes  Lee’s 
advice  that  meeting  management  is  like  program  management,  so  it  states  that  technical 
review  boards  should  be  approached  and  managed  like  a  project.  My  personal  program 
management  experience  has  been  that  you  always  have  limited  resources  and  time  to  perform 
the  required  objectives  on  any  program  and  a  statement  was  included  to  the  effect. 

The  next  part  of  the  section  III  is  based  on  personal  experience  where  the  TRB 
chairperson  should  consult  with  several  people  prior  to  conducting  TRB  meetings.  The 
AFRL-West  Test  Lead  and  the  responsible  branch  chief(s)  are  the  immediate  management 
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authority  the  TRB  chairperson  is  chartered  by,  and  it  is  wise  for  the  TRB  chairperson  to 
make  sure  that  their  concerns  are  addressed  as  part  of  the  review.  The  TRB  chairperson 
should  consult  the  Safety  Review  Board  (SRB)  chairperson  to  work  out  a  plan  synergizing 
the  reviews  as  much  as  possible.  For  example,  the  TRB  may  be  reviewing  a  document  for 
technical  accuracy  and  if  they  include  an  extra  few  minutes  looking  at  safety  criteria  as 
requested  by  the  SRB  chairperson  it  could  save  hours  of  reviewing  the  same  document  in 
detail  later.  Experience  has  shown  that  the  amount  of  total  time  a  program  goes  through  the 
TRB/SRB  process  is  less  if  the  TRB  and  SRB  chairpersons  are  working  together. 
Surprisingly,  not  a  lot  of  TRB  chairpersons  ask  the  program  manager  or  test  director  if  they 
have  specific  concerns  they  would  like  the  review  board  to  address,  which  they  often  do. 
Finally,  the  chairperson  has  to  be  familiar  with  all  the  review  board  member’s  area  of 
expertise  in  order  to  plan  the  review,  and  therefore  should  talk  to  any  TRB  member  they  are 
not  familiar  with  prior  to  making  the  plan  of  who  should  review  each  document. 

The  next  three  paragraphs  relate  to  the  checklist  on  page  4  of  the  guidance  and  the 
TRB  chairperson,  creating  a  specific  list  of  topics  for  the  TRB  to  review  for  the  test  program. 
The  checklist  itself  will  be  discussed  later  in  this  section.  The  TRB  chairperson  will  gather  a 
list  of  all  the  topics  obtained  from  consulting  as  previously  described  and  what  he  decided  to 
include  from  the  checklist.  It  is  important  for  the  chairperson  to  make  sure  the  review  board 
membership  contains  all  the  expertise  required  to  provide  a  good  review  and  this  is  explained 
in  the  guidance.  Finally,  the  chairperson  takes  the  list  and  creates  a  preliminary  plan  of  who 
will  review  each  document. 

The  current  practice  for  TRBs  is  for  the  review  board  to  always  review  the 
procedures,  but  there  is  no  consistence  to  what  is  reviewed  other  than  the  procedures.  Some 
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TRBs  have  solely  reviewed  procedures,  while  other  boards  have  reviewed  procedures  and 
one  or  two  analyses,  and  still  other  boards  have  reviewed  procedures,  analyses,  and  the 
system  design.  Any  of  the  reviews  listed  in  the  last  sentence  may  be  acceptable  as  long  as 
the  reason  for  what  is  reviewed  has  been  thought  out.  Some  TRB  chairperson  request 
specific  documents  from  the  test  team  to  review,  but  frequently  the  TRB  reviews  what  the 
test  team  decides  to  provide  them.  If  section  III  is  followed  by  a  TRB  chairperson  he  will  be 
better  prepared  to  perform  a  comprehensive  review  of  topics,  as  the  choice  of  what  is 
reviewed  will  have  been  thought  out. 

Section  IV  First  TRB  Meeting 

Section  four  explains  what  is  expected  at  the  kickoff  meeting.  The  test  team  will 
present  an  overview  of  the  program  and  the  schedule;  this  is  a  current  best  practice.  The 
TRB  chairperson  will  present  his  plan  and  request  feedback  from  the  test  team  and  TRB 
members.  The  request  for  feedback  on  the  plan  follows  the  theme  of  Staren’s  comments 
about  requesting  input  on  the  agenda  from  the  members  and  Lee’s  comments  about  the 
members  wanting  to  contribute.  The  TRB  chairperson  presenting  a  plan  for  the  reviews  and 
requesting  feedback  will  be  new  to  the  TRB  process.  This  will  allow  the  test  team  and  TRB 
members  to  provide  comments  and  plan  their  time  and  workload  around  TRB  meetings. 

Section  V  Document  Review 

The  first  paragraph  starts  with  emphasizing  the  purpose  of  the  TRB,  which  is  to 
assess  the  technical  soundness  and  adequacy  of  the  approach  to  meet  the  test  requirements. 
Then  it  states  the  TRB  can  also  make  recommendation  for  better  designs  and  ways  to  meet 
the  test  requirements.  This  is  followed  by  a  warning  against  treating  recommendations 
based  on  review  board  member  preferences  as  if  they  are  requirements  the  test  team  must 
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meet.  Examples  are  then  included  to  clarify  this  statement.  This  paragraph  in  the  TRB 
document  was  created  using  several  of  the  lessons  from  the  literature  review  in  mind, 
combined  with  my  own  personal  experiences.  Both  the  test  team  and  TRB  members  have  to 
be  actively  involved  in  the  process  for  it  to  work,  which  follows  Swiss’s  remark  (1992), 

“. . .  .quality  improvement  requires  strong  worker  participation.”  (p.  358)  This  paragraph 
provides  boundaries  for  what  should  be  a  TRB  technical  risk  versus  a  recommendation. 

TRB  technical  risks  will  not  be  ignored  by  AFRL  management,  thus  requiring  the  test  team 
to  address  them.  TRB  recommendations  are  advice  given  to  the  test  team,  but  the  decision  is 
left  to  the  test  team.  Having  clear  roles  and  responsibilities  allows  both  the  TRB  members 
and  the  test  team  to  make  a  contribution  and  take  pride  in  their  work.  Tompkins  stressed  one 
of  Deming’s  14  points,  that  pride  in  your  work  is  a  major  motivator  and  barriers  to  being  able 
to  take  pride  in  workmanship  should  be  removed,  which  was  incorporated  by  the  inclusion  of 
clear  roles  and  responsibilities. 

The  second  paragraph  explains  why  not  all  documents  require  the  same  level  of 
review.  Then  it  gives  a  few  examples  and  ends  with  a  statement  that  the  level  of  review 
should  be  increased  if  major  flaws  in  the  document  are  found.  This  paragraph  incorporates 
the  principles  of  Swiss’  comments  on  TQM  being  opposed  to  mass  inspection  because  they 
shift  the  responsibly  away  from  the  designers.  It  is  important  for  the  test  team  to  take  the 
lead  on  the  program  and  not  the  review  board,  and  therefore  the  review  should  only  be  what 
is  required  to  complete  the  task. 

The  peer  review  concept  in  paragraph  three  comes  from  Harkins’  statements  about 
having  a  detailed  technical  review  prior  to  a  design  review.  This  paragraph  and  paragraph 
four  are  also  written  in  the  spirit  of  not  shifting  the  responsibility  away  from  the  designers.  It 
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further  continues  the  concept  of  clearly  separating  the  TRB  member  and  test  team  roles, 
allowing  both  groups  to  take  pride  in  their  work.  Paragraph  four  provides  warnings  against 
the  TRB  losing  its  objectivity  and  independence  by  taking  the  test  team’s  responsibilities  on 
as  their  own.  This  is  done  with  the  best  of  intentions  and  desire  to  help  the  test  be  successful. 
The  paragraph  ends  by  providing  the  TRB  chairperson  with  a  way  to  get  the  test  team 
assistance  if  needed,  without  compromising  the  TRB  process  itself. 

The  final  paragraph  of  this  section  proposes  significant  changes  to  the  current  TRB 
meeting  practices.  These  changes  are  based  on  the  literature  review  and  personal 
experience.  First,  it  takes  the  advice  of  Allen  et  al.  and  emphasis  not  meeting  if  the  topic  can 
be  handled  with  E-mails.  The  TRB  members  get  the  document  and  provide  comments  by  E- 
mail,  and  meetings  are  only  held  if  there  are  questions  about  the  comments.  My  experience 
has  been  that  75-90%  of  the  comments  received  by  TRB  members  don’t  require  a  discussion 
and  can  be  accepted  and  inputted  into  the  document  upon  recent.  The  TRB  is  then  only 
meeting  to  answer  the  test  team’s  questions  or  once  a  revised  document  with  the  first  round 
of  comments  have  been  incorporated.  This  way  the  TRB  meetings  deal  only  with 
clarifications,  evaluation  of  the  test  team  alternate  approaches  or  explanations  of  why  TRB 
comments  were  not  incorporated  by  the  test  team.  I  believe  this  will  be  a  significantly  more 
efficient  way  to  perform  TRBs.  This  paragraph  ends  with  a  warning  about  doing  a  group 
procedure  step-by-step  walk  through  of  the  procedures,  as  this  is  a  common  problem  of  many 
TRBs. 

Section  VI  Verification  and  Validation  Plan  (V&V  Plan) 

The  verification  and  validation  plan  is  one  of  the  most  important  and  misunderstood 
documents  that  the  TRB  should  evaluate.  The  V&V  plan  is  the  test  team’s  approach  to 
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ensuring  the  technical  success  of  the  program  and  in  reducing  the  technical  risks.  The  risk 
mitigation  portion  of  the  safety  process  is  well  defined  and  understood  by  test  personnel  at 
AFRL.  Therefore,  it  was  useful  to  include  analogies  to  the  safety  risk  mitigation  process  to 
assist  in  the  understanding  of  technical  risk  mitigation  process.  The  safety  review  process 
uses  a  hazard  analysis  to  document  how  the  safety  risks  on  the  program  will  be  mitigated. 

The  safety  review  board  spends  a  lot  of  effort  on  evaluating  the  hazard  analysis.  The  V&V 
plan  is  the  technical  equivalent  to  the  hazard  analysis,  as  it  documents  how  the  technical  risk 
will  be  mitigated  through  inspection,  analysis,  or  functional  checkouts.  The  TRB  is  to 
evaluate  if  the  test  team  follows  the  V&V  plan  will  that  lead  to  a  high  level  of  confidence  of 
technical  success. 

Requirements  for  the  program  come  from  multiple  sources  and  are  translated  into  the 
V&V  plan.  The  V&V  plan  is  a  complex  checklist,  as  it  contains  requirements  and  then  what 
steps  will  be  done  to  verify  those  requirements  are  met.  This  is  where  Deming’s  lesson  in 
implemented  pertaining  to  the  fact  that  quality  only  has  meaning  if  it  takes  into  account  the 
customer’s  needs.  The  test  plan,  which  is  specifically  spelled  out  as  a  source  of 
requirements,  is  the  customer’s  test  document,  explaining  their  test  requirements.  But  this  is 
a  very  narrow  view  of  the  customer,  as  the  test  team  must  also  satisfy  the  requirements  of  Air 
Force  management,  whose  requirements  are  documented  in  regulations,  policies,  best 
practices,  etc.  Hoyer  and  Hoyer  stated  (2001),  “Somehow,  someone  must  know  what  the 
requirements  are  and  be  able  to  translate  those  requirements  into  measurable  product  or 
service  characteristics”  (p.55)  The  first  step  in  creating  the  V&V  plan  is  to  translate  the 
broad  view  of  customer  requirements  into  something  that  is  definable.  The  last  sentence 
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defines  what  the  TRB  should  expect  to  find  in  the  V&V  plan  and  it  includes  the  lessons 
Chernak  documents  from  the  software  industry. 

Section  VII  Meeting  Management 

Landale  expressed  that  it  is  important  for  a  meeting  be  efficient,  have  action  oriented 
objectives,  and  for  the  attendees  to  come  prepared.  Landales  lessons  are  incorporated  into 
the  first  two  paragraphs  of  the  section  VII  of  the  guidance. 

Allen,  Rogelberg,  and  Scott’s  work  was  the  inspiration  for  much  of  the  next  two 
paragraphs.  The  third  paragraph  was  based  on  their  concept  of  allowing  employees  who  are 
part  of  the  group  to  opt  out  of  meetings  that  don’t  require  them.  An  opt  out  clause  was 
included  in  the  third  paragraph.  The  fourth  paragraph  consists  of  their  questions  about  when 
to  have  a  meeting  turned  into  criteria  of  do  not  meet  if  and  reworded  to  reflect  local  practice. 
The  fourth  paragraph  states  meetings  should  not  be  held  if  E-mail  can  be  used  to  deal  with 
the  topic,  there  is  no  expected  outcome  or  action  items  from  meeting,  or  the  documents 
required  for  review  were  not  sent  out  in  advance. 

Several  authors  commented  on  the  importance  of  an  agenda,  and  paragraphs  five  and 
six  plus  the  sample  agenda  on  page  five  of  the  handbook  section  incorporate  their  comments. 
The  inclusion  of  date,  time,  location,  purpose  of  the  meeting,  status  updates,  discussion 
topics,  and  topics  for  future  meeting/schedule  were  inspired  by  Kloppenborg  and  Petrick’s 
work.  The  required  attendees  is  an  extension  of  Allen,  Rogelberg,  and  Scott’s  opt  out  clause 
by  taking  it  a  step  further  and  letting  TRB  member  know  in  advance  that  they  don’t  have  to 
be  at  that  particular  meeting  if  they  are  not  required.  Additionally,  having  the  agenda  sent 
out  early  allows  the  TRB  members  to  opt  out  once  they  know  the  topics.  The  inclusion  of 
required  meeting  preparation  was  based  on  the  statements  of  Landale,  Staren,  Lee,  Allen, 
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Rogelberg,  and  Scott  on  the  importance  of  coming  to  the  meeting  prepared.  Staren’s  advice 
to  prepare  an  agenda  and  send  it  out  early  to  allow  people  to  give  feedback  and  to  plan  is 
incorporated. 

The  final  paragraph  on  meeting  minutes  and  the  sample  meeting  minutes  on  page  6  of 
the  handbook  section  were  based  mostly  on  the  works  of  Staren,  Lee  and  Lazarus  with 
adaptation  to  meet  local  practice.  The  statement  that  the  meeting  minutes  should  be  sent  out 
within  two  working  days  reflects  Staren’s  statement  about  minutes  being  sent  out  in  a  timely 
manner.  The  requirement  for  new  action  items,  responsible  parties,  and  due  date  came  from 
Lee  and  Lazarus’  writings. 

Checklist  of  Potential  TRB  Topics  to  Review 

The  checklist  is  divided  into  three  columns.  The  first  column  is  potential  TRB  topics, 
which  have  been  divided  into  categories  and,  subdivided  further.  The  recommended  level  of 
review  is  listed  in  the  second  column  and  gives  an  indication  of  how  many  or  what  type 
person  should  review  the  document.  In  some  cases  column  two  states  that  only  a  sample  of 
the  documents  need  to  be  reviewed.  The  final  column  is  for  the  TRB  chairperson  to  mark 
which  topics  will  be  reviewed,  with  notes  at  the  bottom  of  the  checklist.  Gawande’s  advice 
on  what  was  a  good  or  bad  checklist  was  incorporated  into  this  checklist,  which  is  why  it  is 
precise,  to  the  point,  and  practical.  Since  this  is  a  new  checklist,  implementation  is  only  a 
first  step.  The  checklist  will  require  updates  to  optimize  it  usefulness  based  on  TRB  member 
inputs. 
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CHAPTER  5 
Conclusion 

Summary  and  Conclusions 

AFRL  has  a  long  proud  history  of  successful  rocket  testing,  although  there  are  enough 
examples  of  test  that  have  gone  wrong.  The  technical  difficulties  and  hazards  related  to 
rocket  testing  require  taking  extra  steps  to  make  sure  all  goes  well.  The  technical  and  safety 
review  boards  are  used  for  this  purpose.  With  a  rapidly  changing  workforce  and  new 
challenges,  it  has  become  more  important  than  ever  to  make  efficient  use  of  all  available 
resources,  especially  personnel.  Current  TRB  reviews  are  inefficient  and  of  inconsistent 
quality.  The  guidance  created  by  this  study  can  assist  the  technical  review  board  chairperson 
in  running  a  more  efficient  and  effective  review. 

The  guidance  incorporates  lessons  learned  about  complexity  of  organizational 
decision  making,  policies,  procedures,  checklist,  organizational  cultural  change,  quality 
assurance,  and  meeting  management;  based  both  on  experience  and  a  thorough  literature 
review.  The  checklist  created  as  part  of  this  guidance  will  assist  in  making  sure  the  TRB 
chairperson  does  not  overlook  a  critical  topic  for  review.  Following  the  advice  of  Gawande, 
the  checklist  is  clear,  concise,  and  practical. 

Recommendation  1  -  Adoption  of  the  TRB  Guidance 

The  Space  Missile  Propulsion  Division  should  adopt  the  TRB  guidance  following  the 
standard  handbook  guidance  review  process.  The  guidance  as  it  appears  in  appendix  B 
should  be  treated  as  the  first  draft.  This  draft  would  then  be  distributed  to  a  group  of 
experienced  test  directors  from  the  Experimental  Demonstration  Branch  for  review  and 
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comment.  Once  their  comments  are  incorporated,  the  revised  draft  would  be  distributed  for 
review  and  comment  to  a  group  including  safety,  quality  assurance,  and  personnel  from  other 
branches  that  perform  rocket  testing.  The  final  product,  with  comments  incorporated,  will 
then  be  submitted  to  management  for  approval.  Once  approved  the  guidance  will  be 
processed  by  the  Configuration  Management  Office  and  posted  on  the  configuration 
management  drive  and  in  the  procedure  management  system.  This  guidance  will  be 
reviewed  and  updated  with  inputs  from  the  first  few  TRBs.  After  that  the  guidance  will  be 
reviewed  periodically  to  detennine  if  updating  is  required. 

TRB  chairpersons  appointed  from  the  time  of  adoption  should  receive  a  copy  of  the 
guidance  prior  to  their  first  time  serving  as  a  TRB  chairperson.  Additionally,  a  first  time 
TRB  chairperson  should  be  assigned  an  experienced  TRB  chairperson  as  a  mentor.  It  has 
been  my  experience  that  a  mentor  is  almost  always  available,  but  few  seem  to  ask  for  one.  If 
management  were  to  inform  a  new  TRB  chairperson  that  a  mentor  is  available  most  would 
consult  with  one  or  at  the  very  least  appreciate  knowing  one  is  available. 

Recommendation  2  -  Create  a  Similar  Safety  Review  Board  Guidance 

Safety  Review  Boards  and  Technical  Review  Boards  are  similar  processes  just  with 
different  focuses.  Guidance  for  conducting  a  safety  review  would  be  almost  as  valuable  as 
guidance  for  conducting  a  technical  review.  The  only  reason  TRB  guidance  is  critical  is 
because  there  is  a  larger  pool  of  TRB  chairpersons  while  only  a  small  pool  of  SRB 
chairpersons,  as  SRB  chairpersons  can  only  come  from  the  safety  office.  In  current  practice, 
the  SRB  process  is  far  more  standardized  than  TRB  process.  Despite  this  fact,  the  SRB 
process  can  still  see  significant  improvement  from  the  creation  of  guidance  for  conducting  an 
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SRB.  Additionally,  the  guidance  would  be  useful  for  the  training  of  new  safety  personnel 
and  helpful  for  review  board  and  test  team  members  unfamiliar  with  the  SRB  process. 

Another  reason  to  create  SRB  guidance  is  to  standardize  the  processes  as  much  as 
possible  between  the  TRB  and  SRB.  A  common  practice  is  to  have  a  combined  TRB/SRB 
board.  The  combined  board  still  has  two  chairs,  to  make  sure  both  technical  and  safety  are 
evaluated,  but  it  means  the  membership  is  identical  for  the  TRB  and  SRB.  Any  progress 
made  towards  efficiency  in  the  TRB  could  be  negated  if  the  SRB  continues  with  the  previous 
less  efficient  process.  An  even  greater  efficiency  could  be  gained  by  having  the  SRB  follow 
a  similarly  efficient  process. 

Part  of  my  job  after  the  completion  of  this  study  will  be  to  work  with  the  safety  office 
to  create  a  similar  guidance  for  SRBs.  Once  the  SRB  guidance  is  created  there  is  the 
potential  that  the  TRB  and  SRB  guidance  could  even  be  combined  for  greater  efficiency. 

Recommendation  3  -  Creation  of  Verification  and  Validation  Plan  Guidance 

As  expressed  in  the  findings,  the  verification  and  validation  plan  is  one  of  the  most 
important  and  most  misunderstood  documents.  Much  of  the  misunderstanding  of  the  V&V 
plan  comes  from  the  misperception  that  it  is  just  another  document  to  create,  when  it  is 
instead  the  test  teams  plan  for  mitigating  the  technical  risks  of  the  program.  Several  times 
during  this  study  I  had  to  stop  and  refocus  as  I  was  starting  to  create  V&V  plan  guidance 
instead  of  guidance  for  conducting  a  TRB.  This  guidance  is  needed  to  assist  the  test  team  in 
reducing  risks  associated  with  testing  and  the  creation  of  the  V&V  plan  document.  V&V 
plan  guidance  will  also  result  in  a  better  TRB  review,  as  the  same  information  needed  to 


create  the  document  will  assist  in  the  review  of  the  document. 
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Rocket  testing  at  a  research  and  development  site  will  always  have  new  types  of 
technical  risks  associated  with  it,  and  therefore  require  new  ways  to  verify  the  risks  are 
reduced.  The  vast  majority  of  the  technical  risks  associated  with  rocket  testing  are  something 
the  AFRL  test  community  has  seen  and  knows  how  to  handle.  Many  of  the  new  technical 
risks  can  still  be  reduced  by  standard  mitigation  techniques  or  slightly  modified  versions  of 
them.  Having  guidance  for  the  creation  of  the  V&V  plan  that  explains  standard  mitigation 
techniques  and  what  they  are  used  for  will  result  in  lower  risks  on  test  programs. 

Part  of  my  job,  but  not  part  of  this  thesis,  is  to  take  what  was  learned  from  this  study, 
best  practices,  and  personnel  experience  and  create  a  test  operations  handbook  section 
explaining  how  to  create  a  V&V  plan.  This  guidance  will  include  a  description  of  the 
purpose  of  many  of  the  most  common  checkouts  performed. 

Research  Recommendation 

As  pointed  out  earlier,  the  TRB  and  SRB  processes  are  similar  just  with  different 
focuses.  One  element  of  the  SRB  process  that  is  well  defined  is  the  risk  matrix  for  rating 
safety  risks  as  high,  medium,  or  low.  This  risk  rating  is  quantified  by  level  of  harm  to  a 
person  or  dollar  amount  of  damage  to  the  facility  combined  with  a  probability  of  occurrence. 
A  risk  rating  is  given  to  the  hazard  prior  to  mitigation  and  then  another  after  risk  mitigation 
has  been  implemented.  For  technical  risks  there  is  no  equivalent  to  the  safety  risk  matrix.  A 
good  technical  risk  matrix  with  a  qualifying  risk  assessment  of  high,  medium,  or  low  would 
greatly  assist  and  enhance  the  TRB  process.  Future  research  will  be  performed  to  create  a 
good  technical  risk  matrix. 
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Conducting  a  Technical 
Review  Board 

I  OPR 

AFRL/RZSO  (Mr.  Dan  Harbour) 

II.  GENERAL  INFOMATION 

The  purpose  of  this  guide  is  to  provide  a  common 
base  for  conducting  technical  review  boards  (TRBs) 
at  the  Air  Force  Research  Laboratory  (AFRL). 
Edwards  Research  Site  for  rocket  test  programs. 
This  guidance  applies  to  test  activities  requiring  an 
AFRURZ-W  Form  27.  Expenmentai/Test  Operation 
Safety  Permit- 

III.  PRIOR  TO  THE  FIRST  TECHNICAL  REVIEW 
BOARD  MEETING 

Technical  Review  Boards  should  be  approached 
and  managed  like  a  project  Just  like  any  project 
there  is  an  objective  to  meet  schedule,  and 
resources  are  limited. 

Upon  appointment  a  TRB  chairperson  should 
consult  the  following: 

•  The  AFRL- West  Test  Lead  and/or 
responsible  branch  chiefls)  to  find  out  if 
there  are  any  specific  topics  or  concerns  she 
has  that  should  be  addressed  by  the  TRB. 

•  The  SRB  chairperson  in  order  to  work  out  a 
plan  to  allow  for  a  smooth  transition  from 
TRB  to  SRB 

NOTE:  All  references  to  a  transition  from 
TRB  to  SRB  assume  the  two  review 
boards  are  separate.  If  the  review  boards 
are  combined  then  the  same  is  true,  but  it 
aids  in  the  running  of  the  combined 
TRB/SRB. 

•  The  program  manager  and  test  director  to 
find  out  if  there  are  any  specific  topics  or 
concerns  they  would  like  the  TRB  to 
address.  Also,  the  program  manager  and/or 
the  test  director  can  supply  a  list  of 
drawings,  analyses,  and  procedures. 

•  Any  unfamiliar  TRB  member.  It  is  important 
to  know  each  member's  area  of  expertise 


The  next  step  is  to  take  the  checklist  on  page  4  of 
this  document  and  determine  what  topics  should  be 
reviewed  It  is  recommended  to  discuss  this  with 
the  SRB  chairperson  as  it  could  aid  in  the  smooth 
transition  from  TRB  to  SRB 

Now  you  should  have  a  list  of  topics  to  be  covered 
by  the  TRB  from  the  checklist,  which  incorporates 
discussions  with  the  AFRL-West  Test  Lead  and/or 
responsible  branch  chiefs).  SRB  chairperson, 
program  manager,  and  test  director  Review  the  list 
and  compare  the  topics  with  the  review  board 
members  expertise.  If  the  review  board  members 
have  the  expertise  required  to  provide  a  good 
technical  review  then  proceed.  If  the  review  board 
members  do  not  have  all  the  required  expertise  to 
provide  a  good  technical  review,  identify  the  missing 
expertise,  and  request  assistance  from  the  AFRL- 
West  Test  Lead  and/or  responsible  branch  chiefjs) 
in  filing  the  gap.  Subject  matter  experts  can  be 
assigned  to  the  TRB  to  consult  only  on  the  specific 
topics  related  to  their  area  of  expertise. 

Take  the  Kst  of  topics  and  come  up  with  a 
preliminary  plan  of  who  wil  be  reviewing  which 
documents,  and  the  level  of  detail  required  for  each 
review. 

IV.  FIRST  TRB  MEETING 

The  first  TRB  meeting  is  a  kickoff  meeting.  The 
program  manager,  test  director,  and/or  test  engineer 
will  present  an  overview  of  the  program  and  the 
schedule.  The  schedule  will  include  anticipated 
dates  that  documents  wil  be  available  for  the  TRB  to 
review. 

The  TRB  chairperson  wil  discuss  the  list  of  topics 
that  wil  be  covered  by  the  TRB.  Present  his  plan  for 
who  will  be  reviewing  which  documents  The  test 
team  and  TRB  members  will  provide  the  TRB 
chairperson  feedback  on  the  list  of  topics  and  plan. 

V.  DOCUMENT  REVIEW 

The  TRB  is  to  review  and  assess  the  documents  for 
technical  soundness  and  adequacy  to  meet  the  test 
requirements.  The  TRB  can  make  recommendation 
for  better  system  designs  and  ways  to  meet  the  test 
requirements.  It  is  not  the  function  of  the  TRB  to 
make  technical  decisions  for  the  program  or  id 
require  designs  meet  the  board  members  personal 
preferences.  For  example,  a  TRB  determining  that  a 
system  design  is  inadequate  without  a  check  valve 
separating  the  liquid  run  line  from  the  nitrogen  purge 
line  is  what  the  TRB  should  be  presenting  as  a  risk 
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to  the  program  success.  The  TRB  requiring  system 
setup  to  be  performed  in  a  particular  order  due  to 
review  board  members  preference,  and  not  because 
of  a  technical  risk,  would  be  inappropriate.  The  TRB 
could  recommend  the  order  to  be  changed  and 
explain  their  reasoning  to  the  test  team,  but 
decisions  related  to  preferences  should  be  left  to  the 
test  team. 

Not  all  documents  require  the  same  level  of  review. 
For  example,  a  complex  analysis  performed  by  a 
professional  engineer  may  only  require  a  quick 
review  by  a  board  member  or  two  to  make  sure  the 
proper  analysis  was  performed.  While  system 
design  drawings  would  require  several  members  of 
the  board  to  review  it  and  make  recommendations 
based  on  their  area  of  expertise  If  major  flaws  in  a 
particular  document  are  found,  then  the  level  of 
review  should  be  increased. 

The  test  team  shotid  perform  a  peer  review  of  all 
documentation  prior  to  submitting  it  to  the  TRB  The 
TRB  should  not  accept  a  document  without  a  signed 
peer  review  page  by  at  least  two  lest  team  members 
in  addition  to  the  author  If  documents  are  received 
without  the  peer  review  page,  the  TRB  chairperson 
should  not  send  the  document  to  the  TRB  members 
until  the  signed  peer  review  page  is  received. 

The  TRB  should  not  accept  poor  quality  documents 
for  review.  Past  experience  has  shown  this  can 
turn  the  TRB  into  a  design,  analysis,  or  procedure 
writing  group  for  the  program,  wrfiich  defeats  the 
purpose  of  an  independent  technical  review  If  the 
test  team  provides  the  TRB  with  poor  quality 
documentation,  then  the  TRB  chairperson  should 
return  the  document  to  the  test  team  for 
improvement  prior  to  submitting  it  for  TRB  review,  tf 
the  test  team  continually  produces  poor  quality 
documentation,  then  the  TRB  chairperson  should 
request  the  AFRL  West  Test  Lead  and/or 
responsible  branch  chieffs)  provide  the  test 
program  with  someone  with  the  required  expertise  to 
assist 

When  documents  are  submitted  to  the  review 
boards,  the  TRB  chairperson  will  distribute  them  to 
the  TRB  members  that  will  be  providing  the  review  of 
that  document  The  TRB  members  will  E-mail 
comments  to  the  TRB  chairperson,  program 
manager,  and  test  director  A  meeting  will  only  be 
held  if  there  are  questions  about  the  TRB  member's 
comments.  The  program  manager  and  test  director 
will  have  the  document  revised  and  resubmit  <  The 
TRB  members  wiB  review  the  revised  document  and 
then  a  meeting  win  be  held  to  discuss  any  questions 


and  comments  that  were  not  incorporated  by  the  test 
team.  The  TRB  members  sitting  around  the  table 
doing  a  step  by  step  group  walk  through  of  the 
procedures  most  likely  means  either  the  test  team 
has  provided  a  poor  quality  document  the  TRB 
members  have  not  come  to  the  meeting  prepared,  or 
the  TRB  has  elevated  the  level  of  review  and 
decided  to  make  themselves  the  procedure  writing 
group. 

VI.  VERIFICATION  &  VALIDATION  PLAN  (VSV 
plan) 

The  V&V  plan  is  one  of  the  most  important 
documents  for  the  TRB  to  evaluate  The  V&V  plan  is 
the  test  teams  approach  to  ensunng  the  technical 
success  of  the  program  by  reducing  the  technical 
risks.  The  V&V  plan  is  the  technical  equivalent  to 
the  hazards  analysis  used  to  reduce  the  safety  risks 
on  the  program.  The  V&V  plan  lists  the  inspections, 
analysis,  and  functional  checkouts  that  will  be 
performed  to  provide  confidence  the  test  will  be 
successful  The  TRB  is  to  evaluate  whether  the  test 
team's  following  of  the  V&V  plan  will  lead  to  a  high 
level  of  confidence  of  technical  success. 

Requirements  no  matter  the  source  (test  plan, 
regulations,  best  practices,  etc.)  are  translated  into 
the  V&V  plan.  The  V&V  plan  should  include  where 
the  requirements  come  from,  how  to  detect 
compliance,  and  what  type  of  inspection,  analysis, 
checkout  or  combination  of  these  will  be  used  to 
verify  the  systems  are  ready  for  test. 

VII.  MEETING  MANAGEMENT 

Due  to  the  fact  that  the  TRB  members,  including  the 
TRB  chairperson,  are  doing  these  reviews  in 
addition  to  their  normal  program  work,  it  is  critical 
that  TRB  meeting  are  an  efficient  use  of  time. 

TRB  meetings  need  to  have  action  oriented 
objectives.  Everyone  must  come  to  the  meeting 
prepared  and  with  their  out  of  meeting  action  items 
completed. 

Only  the  TRB  members  required  to  cover  the  topics 
for  that  meeting  should  be  at  the  meeting.  TRB 
members  should  be  allowed  to  miss  the  meeting  if 
the  topics  being  covered  are  not  related  to  their  area 
of  expertise. 
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Meeting  should  not  be  held  if  any  of  the  following  are 
true. 

•  The  topic  can  be  dealt  with  through  E-mails. 

•  No  expected  outcome  or  action  items  would 
result  from  the  meeting  Don't  hold  regularly 
schedule  TRB  meetings  just  because  the 
scheduled  day  comes. 

•  The  documents  for  review  were  not  sent  out 
in  advance  to  allow  TRB  members  to 
prepare  for  the  meeting 

It  is  important  for  each  TRB  meeting  to  have  a  well 
organized  and  thought  out  agenda  The  agenda 
should  start  with  the  most  critical  topics.  (See 
sample  agenda  on  page  5  of  this  document)  The 
following  should  be  included  in  the  agenda 

•  Meeting  Date.  Time  and  Location 
■  Purpose  of  toe  Meeting 

•  Required  Attendees 

NOTE:  Not  all  members  are  required  at 
all  meetings.  Only  those  with  the 
expertise  required  for  that  meeting's 
topics  should  attend. 

•  Required  Meeting  Preparation 

•  Status  Update  on  Action  Items 

NOTE:  List  only  those  that  will  be 
discussed  at  that  meeting 

•  Discussion  Topics 

•  Topics  for  Future  MeetingsiSchedule: 


The  agenda  should  be  prepared  and  sent  out 
enough  to  advance  for  TRB  members  and  test  team 
members  to  provide  feedback  Doing  tors  allows  the 
agenda  to  be  used  for  planning  and  preparation. 
TRB  members  can  then  inform  the  TRB  Chairperson 
if  the  Sst  of  topics  does  or  does  not  fit  their  area  of 
expertise.  Additionally,  it  allows  TRB  or  test  team 
members  to  inform  the  TRB  chairperson  if  an  action 
item  is  not  going  to  be  completed  in  time  for  that 
meeting's  discussion  or  if  there  are  other  schedule 
changes. 

Meeting  minutes  should  be  sent  out  within  two 
working  days  after  toe  meeting.  (See  sample 
meeting  minutes  on  page  6  of  this  document) 
Meeting  minutes  should  include  the  following 

•  Meeting  Date  and  Time 

•  List  of  Attendees 

•  New  Action  Items.  Responsible  Parties  and 
Due  Date 

•  Action  Item  Status  Updates 

■  Decisions  Made 

•  Topics  for  Future  Meetings/Schedule 
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Checklist  of  Potential  TRB  Topics  to  Review 


Potential  TRB  Topics 

Recommended  Level 
of  Review 

TRB 

Review 

Design  Drawings 

P&IDs 

Multidiscipline  Review 

Isometrics 

Single  Person  Review 

Rigging  Drawings 

One  or  Two  Person  Review 

Electrical  Drawings 

EE,  Electrician,  or  IT 

Layout  Drawings 

One  or  Two  Person  Review 

Other  Drawings 

Varies  Depending  on  the  Drawing 

System  Design 

Component  Specifications 

Single  Person  Review  of  a  Sample  of 
the  Documentation  Available 

Material  Compatibility 

Single  Person  Review  of  a  Sample  of 
the  Documentation  Available 

Remote  Valve  Fail  Safe  Position 

Single  Person  Review  of  a  Sample  of 
the  Documentation  Available 

Pressure  Rating 

Single  Person  Review  of  a  Sample  of 
the  Documentation  Available 

Code  Compliance 

Single  Person  Review  of  a  Sample  of 
the  Documentation  Available 

Cleaning  Specifications 

Single  Person  Review 

Instrumentation  and  Controls 

Single  Person  Review 

Other  Design  Documents 

Varies  Depending  on  the  Document 

Analysis’ 

Structural 

One  or  Two  Person  Review 

Flow 

One  or  Two  Person  Review 

Thrust  Stand 

One  or  Two  Person  Review 

Relief  Valve  Sizing 

One  or  Two  Person  Review 

Lift  Plan 

One  or  Two  Person  Review 

Waterhammer 

One  or  Two  Person  Review 

Other  Analysis 

Varies  Depending  on  the  Analysis 

Procedures 

T est  Operating  Procedures 

Multidiscrpline  Review 

Standard  Operating  Procedures 

Mu  Indiscipline  Review 

Verification  and  Validation  Plan  (V&V  Plan)** 

Multidiscipline  Review 

Non-Conformance  and  Corrective  Action  Report 
(NCAR) 

Varies  Depending  on  the  Non- 
Conformance 

NOTES: 

*The  analyses  to  be  performed  are  listed  m  the  V&V  Plan,  but  are  spelled  out  separately  here.  The  TRB 
should  not  just  review  what  analyses  are  planned,  but  also  the  analysis  itself. 

**The  V&V  Plan  will  include  a  list  of  the  analysis,  testmg'demonstration  (functional  checkouts),  and 
inspections  that  will  be  performed  as  part  of  the  program. 
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Sample  TRB  Agenda  for  Program  X 


Meeting  Date:  25  July  201 1 


Time:  8:00  am 


Location:  Building  8351.  Room  230 


Purpose:  To  provide  final  comments  on  the  documentation  required  for  the  X  motor  receiving  and 
installation  into  the  test  pad.  Status  igniter  system  design  review. 


Required  Attendees:  George  Washington.  John  Adams,  etc 


Required  Meeting  Preparation:  Review  the  rigging  drawings,  lift  plan,  motor  move  and  installation 
procedures  revisions  and  be  prepared  to  answer  test  teams  questions  about  TRB  member  comments 
and  or  discuss  TRB  member  comments  that  the  test  team  did  not  include  in  the  revision. 


Status  Update  on  the  Following  Action  Items:  Responsible  Parties: 

Igniter  system  design  review  Thomas  Jefferson,  Janies  Madison 


Discussion  Topics: 

Rigging  Drawings 
T  ift  Plan 

Motor  Move  Procedure 

Motor  Installation  Procedure 

Test  director  to  present  modified  plan  for  xxxxxx 


Topics  for  Future  Meetings' Schedule: 
Test  Plan 
V&V  Plan 
TOP 
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Sample  TRB  Meeting  Minutes  for  Program  X 


Meeting  Date:  25  July  201 1 


Time:  8:00  am 


Location:  Buildmg  8351,  Room  230 


Attendees:  George  Washington.  John  Adams,  etc 


New  Action  Items: 

Review  igniter  checkout  portion  of  V&V 


Responsible  Parties:  Due  Date 

Thomas  Jefferson,  James  Madison  2  Aug  1 1 


Action  Items  Status  Updates: 
Igniter  system  design  review 


Responsible  Parties:  Due  Date: 

Thomas  Jefferson,  James  Madison  Complete 


Decisions  Made: 

Rigging  drawing,  lift  plan,  and  motor  move  procedures  are  ready  for  signature. 
Motor  installation  procedure  is  ready  for  signature  once  s  has  been  added 


Topics  for  Future  Meetings' Schedule: 
Test  Plan  -  2  Aug 
V&V  Plan -2  Aug 
TOP-  9  Aug 
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